Why Not One Application Per Server / Domain?
A WebArchive (WAR) deployment in Java EE 6 is productive (you have to zip only once), simple (there is only one project in the IDE) and powerful enough for most use cases. However, with an Enterprise Archive (EAR) you can group multiple WARs together and deploy them at once as a logic unit--an application.
There is no hierarchical WARs, so you will have to invent some naming conventions to express application membership.
Usually you would like to have the highest possible isolation between applications. The easiest possible approach is to use an entire server to host a single application. A concept of domains (in Glassfish case) or logical servers would suffice as well. With a domain named after the application and containing only a single application you get additional benefits as well:
- Each server runs in an isolated JVM. Classloader interferences between the applications are impossible.
- You can easily upgrade a single application without affecting the others.
- An application can demand more heap, than on a shared server.
- CPUs / cores are better utilized. Multiple JVMs scale better than a single one.
- A server per application is easier to manage: there is clear relation of monitoring data, logs and bandwidth to an application
- Throttling is easier to implement.
- A single application can be deployed in multiple versions at once.
- If one application dies, the others will still run.
- A security of one application not necessarily has to affect the others.
There is of course also a drawback: an application server instance comes with ~100MB overhead. With the deployment of ten applications comes about 1 GB of wasted RAM (~50 Euros :-))
See you at Java EE Workshops at MUC Airport!
Thanks for this - I've been recently mulling over whether to use Virtual Servers or separate domains in Glassfish and this makes it a no-brainer :)
Posted by Phill on March 15, 2013 at 10:39 AM CET #
I totally agree on the listed benefits of different JVMs per application. Nevertheless, there may be a second drawback (for some peoples): License cost of the application server. If you are forced by company policies to use a specific application server, you could easily end in doubling you total run costs with 3 or 4 applications. Of course, yes than the company want it this way but the problem is that the management next to me has different opinion about spending money that the company IT governance.
Posted by karschdn on March 15, 2013 at 11:38 AM CET #
Well, it may be an option. Ideally, app servers should be able to handle this without having to keep a separate copy around.
I like to structure my webapps into a lot of fairly small web services. Having been through pretty much all ways of structuring applications, this has turned out to be one of the best. The only downside is quite a lot of duplication for deployment (separate WARs - huge EARs with shared libs always give me classloading issues).
But I wouldn't want to have dozens of JVMs for that. Which is one reason why I don't like Dropwizard (as cool as it looks).
And also note that RAM may be cheap on physical machines, but on cloud machines things can be different.
Posted by Marius on March 15, 2013 at 11:50 AM CET #
Something to invent a Maven archtype for. I've often seen this construct of domains before
Posted by openwms on March 15, 2013 at 12:19 PM CET #
I think mostly this points out the sheer insanity of the whole application deployment model for Java EE in this day and age.
Running multiple applications in the same process made sense when people had 64K of RAM, as an optimization. It makes none today.
If your application server needs 100Mb for itself, it be time to consider that it's a bad technology that you should throw away.
I frequently see people lose days wrestling with application server configuration issues - when the value they're actually getting from using one is very close to zero. In this day and age, people use Java application servers because they were told long ago that they needed one and it never crossed their mind that it wasn't true, or might not be true anymore. That and fashion - it just wouldn't be fashionable or professional or enterprisey if you could run your server with "java -jar" - it's got to be at least a little more complicated than that!
Posted by Tim Boudreau on March 15, 2013 at 12:54 PM CET #
What a joke!
This is typically why most of people find Java infrastructures so heavy and so much memory consumer... Just doing a parallel, would you even dare doing the same on an Apache/NGINX server proxying a Python webapp or a ROR webapp??? No, because this is simply stupid.
Yeah, "1 GB of wasted RAM (~50 Euros :-)" is nothing, but maybe you might eventually find that you run out of memory slots... Would you even think about what you could with this wasted memory to support an higher load? NO, of course not, because there is an alternative: "let's create a cluster of servers, this will make it this time" (and please Oracle send me the invoice for the new required licenses!)
Seriously, initially, Java application servers were supposed to host several applications, protecting them from others (classloader interferences, application reloading that always finish with an OutOfMemoryError...), and be capable of support an huge load. Java infrastructure is FAR away from this, ending with bloated software of several hundred megabytes of JARs, useless technologies like OSGI (because nobody is capable of understanding the current status and libraries involved in the application server, the best is to isolate everything, what a great move!)
Maybe one day you will discover JEE is an enterprise *only* technology, just because marketing teams from Oracle and Service companies did their job pretty nicely. But for sure, it's not a fun and developer friendly technology, just some overbloated and overengineered thing, paying its 15 years of retro-compatibility.
If you are not convinced, just have a look to JSF 2 "JEE standard web framework" and compare it to the real web pattern HTML5/JS/CSS3. Definitely something totally out of the scope of real usage.
Thank you for the good work, Oracle guys, seriously.
Posted by Anonymous on March 15, 2013 at 01:49 PM CET #
1) you must allocate a new set of ports for each instance on the same machine
2) you administer more domains and their configurations are not shared
3) you cannot share pools between applications, if they use the same database it may be a problem.
We still use SGES2.1.1 in production environment, clustered domain, server instances, applications in some groups running on instances with some configurations.
Deploying a new version on production is done in several steps:
- customization of database schema (must be compatible and may be done a night before)
- deploy on admin instance
- lb configuration - users are not directed to first instance
- sync and restart of the first instance
- lb config - users are sent to this instance and not to other instances of the same group
- sync and restart and lb config for the rest of instances
This is done in full service (but in time when the load is lower), only with some warning notice to users that they may experience some problems (which should not occur).
If the first instance works, syncing and starting another is trivial. So we can run several isolated applications, even versions, while still under the same AS domain. The level of isolation is configured as needed ...
Upgrade to GF3 is scheduled to second half of this year ...
Posted by David Matějček on March 15, 2013 at 06:08 PM CET #
"1) you must allocate a new set of ports for each instance on the same machine"
GlassFish comes with CLI feature "portbase" which automatically increases all the ports. Really convenient.
"2) you administer more domains and their configurations are not shared"
Yes, but I usually don't want to share configuration between distinct applications.
"3) you cannot share pools between applications, if they use the same database it may be a problem."
A database can handle thousands of connections. Java EE applications tend to rely on connection pools, so it usually is not a problem.
Thanks for your comment!
Posted by Adam Bien on March 15, 2013 at 06:25 PM CET #
"Well, it may be an option. Ideally, app servers should be able to handle this without having to keep a separate copy around."
The application servers are able to handle multiple applications within a shared domain. Most of them even support OSGi. But why not just 1:1? Is easier and more robust,
thanks for your comment!,
Posted by Adam Bien on March 15, 2013 at 06:26 PM CET #
"What a joke!"
This post was not intended to be a joke. Funny: most of the "heated" comments are anonymous, why?
"Yeah, "1 GB of wasted RAM (~50 Euros :-)" is nothing, but maybe you might eventually find that you run out of memory slots..."
Then I would buy a new machine (recently I did it). If this is not viable, you can still deploy EARs. Cost should not be the reason. Hardware is extremely cheap.
"(and please Oracle send me the invoice for the new required licenses!)"
Why you would like to pay Oracle for TomEE, Jboss, or GlassFish? If you are an Oracle fanboy, you can do this. But usually it is not required to pay licenses for opensource application servers, but a reasonable option.
"Maybe one day you will discover JEE is an enterprise *only* technology, just because marketing teams from Oracle and Service companies did their job pretty nicely."
How do you define "Enterprise"? My blog and heating control are running on GlassFish (two instances of course). The server costs ~1200€ and I could easily start additional 30 instances.
"If you are not convinced, just have a look to JSF 2 "JEE standard web framework" and compare it to the real web pattern HTML5/JS/CSS3. "
Not only I will compare it, but about 30 attendees as well (http://workshops.adam-bien.com/javaee-ui.htm). Take the right tool for the job.
JSF 2 works perfectly with HTML5. Just take a look at: http://lightfish.adam-bien.com
"Thank you for the good work, Oracle guys, seriously."
I'm a freelancer, extremely loosely coupled to Oracle.
thanks Anonymous for the funny comment :-)
So: you would rather like to deploy EARs instead of WARs? :-)
Posted by Adam Bien on March 15, 2013 at 06:36 PM CET #
I'm using application servers because of simplicity. I get everything I need as a single package.
java -jar works as well (even with GlassFish v3), but is a bit more complicated.
100MB were very pessimistic. The overhead of GlassFish, JBoss, TomEE is significantly lower.
thanks for your comment (and also for your NetBeans work),
Posted by Adam Bien on March 15, 2013 at 06:39 PM CET #
Currently I'm working on a project where some of the fundamental goals include modularity, reuse and patchability of the code. To achieve this, instead of simply using one WAR, we have our WAR deployed along with several EJB JARs/Bean Archives. This is deployed to JBoss AS 7. We use JBoss Modules to link everything together into the class loader.
So is this a bad pattern? Sprint to sprint we can update only a subset of JARs/WAR (if needed) to achieve our goal. When we finally move this to production, we can hopefully patch a bug with minimal downtime due to only deploying a single JAR.
Posted by John on March 15, 2013 at 06:57 PM CET #
you could deploy the EJB-JARs with the WAR. Having JARs outside the war would only work for remote communication, or proprietary classloading tricks.
It sounds pragmatic, however, your solution is less portable, than a self-contained WAR,
thanks for the comment!,
Posted by Adam Bien on March 15, 2013 at 10:06 PM CET #
1. Each server runs in an isolated JVM. Classloader interferences between the applications are impossible.
it depends on how you use your application server. if you fill your cp with too many jar with different versions.
yes it is a good idea to split your applications. but most app servers have their own shared library features so it is not a big problem.
2. You can easily upgrade a single application without affecting the others.
this depends how you manage your cluster. if you have multiple domains and lb. you can still upgrade your applications without effecting others.
so this is not a case also.
3. An application can demand more heap, than on a shared server.
yes this is a good reason. but in a shared environment you can ask why you need so much heap. :)
also it is hard to get notify memory leaks. because most of us does not questions "why my application needs large heap?"
4. CPUs / cores are better utilized. Multiple JVMs scale better than a single one.
yes this is also a good reason because it is impossible to fair share cpu and it is hard to find thread locks.
5. A server per application is easier to manage: there is clear relation of monitoring data, logs and bandwidth to an application
Throttling is easier to implement.
if you have a large application cluster. this may have an issue but in that case you probably have a dedicated admins to monitor.
so that is not a case.
6. A single application can be deployed in multiple versions at once.
I think this can not be the reason. becaouse no one have multiple databases , cache servers, session replicated servers.
7. If one application dies, the others will still run.
thats way we have lb and monitoring software.
8. A security of one application not necessarily has to affect the others.
yes this is a good reason.
when you split your application to multiple servers it is very hard to manage all of them. because each application server needs monitoring,
configuration, additional licence fees. licence fees are the most scary part :)
Posted by orhan on March 15, 2013 at 10:42 PM CET #
I'm curious: Assuming you have some sort of virtualization environment, why not skip the multiple JVMs and just have a separate whole VM per application?
Then, your hot-deployment mechanism is to start a new VM and point the DNS record at that one instead of the old one - it's completely decoupled, and if something fails badly, you can instantly switch back to a known-good server.
If you're running something like Linux with the UKSM patchset, you've got usable memory de-duplication - so you're really not paying as much as you might think for having an additional OS image running.
Your application, and the deployment model, can be as simple as it ought to be; and things like updating and throttling are completely decoupled, instead of trying to cram all those concerns into the application process (where it's guaranteed some will be done better and some worse).
That's kind of the thing with virtualization: Any operating system you want to use comes with this great application server - which happens to *be* the operating system.
So does having an application server actually simplify things, or does it just mean working with complexity you're comfortable with?
Posted by Tim Boudreau on March 16, 2013 at 12:50 PM CET #
I don't like your comment--the idea your are describing here was supposed to be my next post :-).
Exactly what I'm doing right now with KVM.
A JVM on bare metal (hypervisor) would be even nicer.
Take a look at: http://www.adam-bien.com/roller/abien/entry/cloudvm_from_waratek_an_interesting
Proprietary, but interesting,
Posted by Adam Bien on March 16, 2013 at 02:45 PM CET #
> Usually you would like to have the highest possible isolation between applications.
Usually you don't have set of completely isolated applications. Sometimes you should upgrade the same JARs for many applications, because you have JMS or some other communication channel, or some sort of ORM and database mapped classes.
> If one application dies, the others will still run.
This is usually not true for related applications. Yes, others will probably run, but can't work alone.
> there is clear relation of monitoring data, logs
Seriously? What about trying to find something in two logs on different hosts? This is relatively complex problem in clustered environment, for example.
In general - your points are valid, but only for some rare use cases.
Posted by Serge on March 16, 2013 at 06:33 PM CET #
"Usually you don't have set of completely isolated applications. Sometimes you should upgrade the same JARs for many applications, because you have JMS or some other communication channel, or some sort of ORM and database mapped classes."
In case the applications are not completely isolated, you can package them together. The idea of the post is to replace a EAR with a set of WARs and deploy everything related to a dedicated domain.
"> If one application dies, the others will still run.
This is usually not true for related applications. Yes, others will probably run, but can't work alone."
If an application is dependent on a service (=application) and the service crashes, the application should still run and show an error. E.g.: this blog will still run in case the stats (x-ray http://x-ray.adam-bien.com) die. You will just don't see the stats...
"Seriously? What about trying to find something in two logs on different hosts? This is relatively complex problem in clustered environment, for example."
But each host emits only data which is closely related to a single application. Interpretation of the data is simpler. Also: if you connect e.g. VisualVM to a 1:1 server you get all the stats of a single application... Also in this case it is easier...
thanks for your thoughts. Regardless what I'm writing in this blog: there is no silver bullet :-)
Posted by Adam Bien on March 16, 2013 at 10:28 PM CET #
yes we have a virtualized environment.
I have hundreds of applications and much much more application servers. before answering questions, I thought can use one jvm for one application? and how can I do that? and it scares me out.
In my opinion there should be some kind of virtual jvms or jvm pools. when i invoke an application. application will switch to a virtual jvm and will run on it. so i can use single application server with different jvm. so anytime I can switch my application to any of them. then we can use on jvm for one application.
Posted by orhan on March 16, 2013 at 10:54 PM CET #
"In my opinion there should be some kind of virtual jvms or jvm pools. when i invoke an application. application will switch to a virtual jvm and will run on it. so i can use single application server with different jvm. so anytime I can switch my application to any of them."
For versioning / upgrades you could also use "share nothing" plus a stock load balancer. For efficiency reasons, I'm not sure whether JVM swapping is the best solution.
Have you looked at: http://www.adam-bien.com/roller/abien/entry/cloudvm_from_waratek_an_interesting ?
Posted by Adam Bien on March 17, 2013 at 06:59 AM CET #
yes I checked your last post. It does not cover my needs. but it is a good start. if someone merges this with jrebel :). this may be an alternative.
Posted by orhan on March 17, 2013 at 04:18 PM CET #
> But each host emits only data which is closely related to a
> single application. Interpretation of the data is simpler.
The logging problem can be solved if you use a consistent and sortable log line format.
Though I think a lot of people didn't understand it (my bad), that's what I was trying to describe here:
I.e. the log format is so simple you can merge them with
cat this.log that.log | sort
so you can look at one process or have an overview of many using simple command-line tools.
Posted by Tim Boudreau on March 18, 2013 at 06:22 AM CET #
this is the correct way to virtualize webapplications http://www.waratek.com/blog/june-2013/open-source-api-for-java-virtualization
now there may be one jvm for each application
Posted by orhan on June 14, 2013 at 01:49 AM CEST #
nice to see that step-wisely (quite) big enterprise requirements are addressed by you JEE experts.
What about building enterprise systems made of multiple ears, whereas each ear is build as you described? Still, there are many issues not covered yet by appserver providers and JEE.
(1) One example is that remote ejb proxies are not automatically recovered after deployment - custom proxy factories need to be implemented.
(3) Another one is the communication between wars ... there is actually no kind web client project type available to provide an API for JSF apps - needed to integrate apps at different WARs (for EJB jars we JEE provides client projects...).
(3) Another aspect ... how to develop single EARs with one or more teams/team members? How to hide parts of sources to subcontractors, but ensure deployability?
From my perspective, development of integrated service landscapes with JEE is still a challenging and not that easy as many hello world example show ... that's why - hope to see more of really real-world enterprise patterns here and also provided by next JEE versions!
Kind regards, Andreas
Posted by Andreas Freier on January 07, 2015 at 03:39 PM CET #
a good question, I'm going to cover it at the next: http://airhacks.tv,
thanks for asking and commenting!
Posted by Adam Bien on January 11, 2015 at 08:01 PM CET #
Thanks for the great article. But I am confused now. Which scenario should I apply for e.g. 25 jee apps?
- one application server and 25 deployed jee apps
- one application and 25 jee apps splited into different app server domains
- 25 application server instances on one vm
- 25 vms, one for each jee app
Posted by tokad on February 24, 2015 at 03:21 PM CET #
I have a doubt about a situation at work. We have a client with a Java web application deployed on a Wildfly 9.0.1 server that runs on a Oracle 1.7.xx JVM.
The Wildfly instance is set up on an over dimensioned physical server, so someone suggested that it could be deployed another web app on the same machine, obviously using another instance of Wildfly as you suggested in this article.
The specific question I have is, there are any differences if we set up each Wildfly instance using different jre (two JRE paths), and using the same JRE path for each Wildfly instance in terms of performance?
Posted by Leonardo on April 06, 2016 at 03:30 PM CEST #