Gaston Dombiak

Scalability: Turn it to Eleven

Blog Post created by Gaston Dombiak Champion on Dec 19, 2006

The parody movie This is Spinal Tap includes a hilarious scene where the members  of the "loudest band in the world" point out that their amplifiers can actually be turned up to eleven. I was reminded of the scene as we've been doing scalability testing on the next version of Wildfire. It turns out we had little idea of how far we could push the limits -- we keep cranking the "scaling knob" louder and Wildfire just keeps scaling. 


So far we've hit 33k concurrent users with a single connection manager, running on a (old) Sun 280R server. CPU usage in the connection manager and core Wildfire server both hovered around 7% each. Those numbers are a pretty huge improvement over the previous version of Wildfire, which was barely able to hit 7500 concurrent users with maxed out CPU and memory usage. We're also only part way through the optimization process. The goal for the 3.2 release is to demonstrate 100k concurrent users on a single domain.


How did we get here? In Wildfire 3.2 we decided to replace our networking layer with Apache MINA, giving us support for asynchronous I/O and a foundation for better scaling. For testing, we created a Wildfire plugin that generates users, populates rosters and creates vCards. The rosters are populated with 20 to 30 contacts each. We've been using the load testing tool Tsung. Tsung is a master-slave tool, and for our tests we are using four slaves.


As we've tested ever higher scalability numbers, we've made lots of core code improvements along the way. For full details, see my forum post. The goal for this week is to reach 50K concurrent users. But based on our experiences over the past couple of weeks, that might not be much of a challenge at all.