Are Rails and Grails Scalable?
Note: Some people keep complaining that I mistook “performance” with “scalability”. Please READ THE ARTICLE! I am talking about the performance *decreases* with increased number of threads. That is *scalability*. Please read before your comment on it again!
The short answer is no and here is why: Graeme Rocher, of the Grails fame, recently posted a simple benchmark comparison between Rails and Grails. While Grails beats Rails in the benchmark, it looks to me neither of them is scalable in a multi-core environment. From our internal tests, JBoss Seam easily out-performs both. 
Let me begin by saying that I admire Graeme’s coding skills — Grails is an excellent framework and JBoss Seam is built on many of the same ideas. But benchmark results like the following really do not bode well with Grails or Rails:
It shows that the overall throughput decreases significantly when Ruby processes increase from 1 to 10, or Tomcat threads increase from 10 to 25. From my own experience in performance testing, that is a tell-tale sign of CPU contention in synchronized code blocks or collection objects. CPU contention does not happen very often on single CPU machines. However, it happens a lot on multi-core (or multi-CPU) systems, which this benchmark is tested on. Basically the CPUs are all waiting to access a single block of synchronized code or object. If you do a JVM thread dump here (or something equivalent in Ruby), you will see most threads are waiting and the CPU load slowly creeps up to 100% as the backlog increases.
Another way to see the CPU contention is to look at application response times. When the CPUs start to wait upon each other, some threads would hang and produce extremely long response time for some requests. So, let’s look at Graeme’s response time graph. The max response times are indeed extremely long — consistent with hang threads. Actually, in this scenario, the mean value of the response time is meaningless since they are screwed by the long response times (i.e., the distribution is no longer normal). The median response time would be a much more robust gauge for user experience here.
Since the benchmark application itself is very simple, it is very unlikely that the contention point is in the application. It has to be in the framework or in the underlying application server. In JBoss, we try very hard to eliminate the contention points in the application server (and in Tomcat). As a result, frameworks like JBoss Seam can easily outperform Grails or Rails based on our internal tests (cannot publish numbers yet. Take my word with a grain of salt! ;)) … That is a huge benefit you get by choosing “mature” application servers.
In any case, I wish the best for Grails. It is probably a minor contention point that show up big time in the CRUD tests. It should be resolved with a simple thread dump (or a profiler). I am sure Graeme will be able to find it and fix it very soon!
March 28th, 2007 at 3:16 pm
Haha, you guys were dying to find something that you could say Seam was better than Grails at. I congratulate you in finding it
I imagine Grails will likely never be as fast as Seam (a pure Java solution), but then one of the worst things you can do in software engineering is optimise early.
The other thing is that Seam is past version uno, we haven’t got to the optimisation part of the dev cycle yet as expressed in the benchmarks
Also please don’t confuse performance and scalability, Seam might outperform Grails, but Grails can scale just like a Seam app can. If performance really is an issue to you for a particular part of the system then you can always fallback to using a Spring MVC controller.
Finally, Grails is not the right choice for every scenario just as Seam isn’t for every scenario. They are different frameworks with different strengths and weaknesses. Choose what is best for you.
March 28th, 2007 at 3:52 pm
Graeme,
I agree 100% with all your points except for the “you guys were dying to find something that you could say Seam was better than Grails at” part. We are not THAT evil.
Also, IMO “scalability” also includes how the framework scales from single-core CPU to multi-core machines. That would require you to fix the concurrency contention points in the app.
cheers
Michael
March 28th, 2007 at 4:54 pm
How many websites process that many dynamic requests per second
Just randomly, wordpress.org - Alexa rank 402
Feb 07 average
81K page views a day / 24 hrs / 60 mins / 60 secs
Almost 0.94 hits per second.
http://www.worldpress.org/traffic.cfm
March 28th, 2007 at 5:56 pm
wgaf,
First of all, you are quoting “average” while a site should really handle “peak” traffic.
Second, keep in mind that the “benchmark” numbers here are for very simple apps with very limited concurrent users. It will be slower when you have a lot of user states to manage.
According to your logic, no one will ever need to use server clusters? In fact, plenty of people do need better performance for all kinds of reasons.
Last, without resolving CPU contention points, the CPU will eventually rise to 100% within several hours and render the site unusable — then you cannot even serve 1 request per second.
cheers
Michael
March 28th, 2007 at 6:53 pm
When will we see some benchmarks for Seam? While there are lots of “proven” marks for Grails, we simply have the 5-10 magic hand wave for Seam
So until you can put up numbers to prove the point isn’t this all somewhat moot?
March 28th, 2007 at 8:55 pm
Uberdude,
I know. That is why I said “take my word with a grain of salt”.
We are not ready to publish because, frankly, Rails/Grails are not our performance objectives. As Graeme said, it is to be *expected* that a pure Java solution will outperform a scripting language based solution. They have different strengths.
cheers
Michael
March 28th, 2007 at 11:01 pm
Certainly I would not expect the current architecture of Ruby on Rails to scale like a Java EE based solution - but I’m not sure that I agree that Grails should not in principle exhibit comparable performance and scalability in simple cases.
After all, Groovy compiles down to Java classes just like ordinary Java. I expect that if the Grails team goes through the full process of identifying contention points and optimizing performance, Grails should scale comparably to Seam at least for simple stateless usecases. (Seam will probably still have the edge in optimistic txn processing, however, due to the conversation model.)
So this is probably more an indication of maturity than of any deep architectural problem in Grails.
March 29th, 2007 at 5:30 am
Please read carefully - my comment was, very few sites handle that number of requests per second. Nothing else
The obversations and assertions you are making are based on a specific traffic scenario that really doesn’t very many web sites.
March 29th, 2007 at 12:32 pm
Graeme is right, scalability and performance are two different things. There is also scaling horizontally, via hardware and vertically via more CPU per hardware. For some scenarios, it would be far cheaper to build quickly, say via Grails, and deploy on cheap hardware and be able to add to that hardware than to build bigger and deploy on bigger hardware.
You also have to take into account the server it is running under. maybe Grails will run better under Glassfish or Jetty or weblogic, or whatever. Some of the request-per-second time is going to be dependent on the servlet engine in use. Not all, but some.
Regardless, until we see a head-to-head match Seam vs. Grails one can only speculate
April 16th, 2007 at 3:00 pm
You are confusing the cost of scaling with the ability to scale. It is more expense to scale Rails and Grails, but they scale just fine.
The choke point is almost always the DB. If you want to scale, make sure that the DB can be scaled (most can’t once you are update bound).
April 16th, 2007 at 4:29 pm
@Uberdude: I don’t have the numbers handy anymore, but my simple seam app running on a 2 year old single proc budget server took a slashdotting, combined with a digging, and a Yahoo tech article linking amazingly well. Obviously that’s no comparison to any other toolset, but personally I was surprised at how well it handled the load.
@wgaf: I haven’t looked where we rank, but Cingular moves orders of magnitude more traffic than that. Scaling beyond those numbers are very important for many sites.
April 17th, 2007 at 7:30 am
Michael with all do respect I think your numbers are too high. Most real apps have greater contention than this. Because Seam/Hibernate supports optimistic locking scenarios and is really optimized for MVCC/Snapshot isolation…I would expect its performance to be at least a full magnitude greater under contention. I think as you mention this mostly tested CPU contention/IO utilization and probably the efficiency (in the case of ruby) of the driver implementation. I would be much more curious to see scalability under contention than these kinds of raw numbers (and of course some hardware to go with it)… Probably you can’t do that on your laptop though
January 14th, 2008 at 12:37 pm
This is an old blog posting i know, but I was wondering if you have thought of retesting this now that Seam 2.0 is out, Rails can run under tomcat (mongrel by a few people I have talked to is a mongrel), and Grails 1.0 is almost out?
Would be interesting to see the changes
ciao