Are Rails and Grails Scalable?
Wednesday, March 28th, 2007Note: Some people keep complaining that I mistook “performance” with “scalability”. Please READ THE ARTICLE! I am talking about the performance *decreases* with increased number of threads. That is *scalability*. Please read before your comment on it again!
The short answer is no and here is why: Graeme Rocher, of the Grails fame, recently posted a simple benchmark comparison between Rails and Grails. While Grails beats Rails in the benchmark, it looks to me neither of them is scalable in a multi-core environment. From our internal tests, JBoss Seam easily out-performs both. 
Let me begin by saying that I admire Graeme’s coding skills — Grails is an excellent framework and JBoss Seam is built on many of the same ideas. But benchmark results like the following really do not bode well with Grails or Rails:
It shows that the overall throughput decreases significantly when Ruby processes increase from 1 to 10, or Tomcat threads increase from 10 to 25. From my own experience in performance testing, that is a tell-tale sign of CPU contention in synchronized code blocks or collection objects. CPU contention does not happen very often on single CPU machines. However, it happens a lot on multi-core (or multi-CPU) systems, which this benchmark is tested on. Basically the CPUs are all waiting to access a single block of synchronized code or object. If you do a JVM thread dump here (or something equivalent in Ruby), you will see most threads are waiting and the CPU load slowly creeps up to 100% as the backlog increases.
Another way to see the CPU contention is to look at application response times. When the CPUs start to wait upon each other, some threads would hang and produce extremely long response time for some requests. So, let’s look at Graeme’s response time graph. The max response times are indeed extremely long — consistent with hang threads. Actually, in this scenario, the mean value of the response time is meaningless since they are screwed by the long response times (i.e., the distribution is no longer normal). The median response time would be a much more robust gauge for user experience here.
Since the benchmark application itself is very simple, it is very unlikely that the contention point is in the application. It has to be in the framework or in the underlying application server. In JBoss, we try very hard to eliminate the contention points in the application server (and in Tomcat). As a result, frameworks like JBoss Seam can easily outperform Grails or Rails based on our internal tests (cannot publish numbers yet. Take my word with a grain of salt! ;)) … That is a huge benefit you get by choosing “mature” application servers.
In any case, I wish the best for Grails. It is probably a minor contention point that show up big time in the CRUD tests. It should be resolved with a simple thread dump (or a profiler). I am sure Graeme will be able to find it and fix it very soon!
