OK, not a billion times, but more like 6 times faster than we were a month ago.

When we first started out, PBwiki was super-fast because there wasn’t much going on. As we’ve grown two things have happened - first, we’ve added lots of useful features and second, we now have lots more traffic all the time. The new editor in particular has led to lots more users being comfortable with editing, and it’s a lot of code to serve up, so we were serving lots more data more frequently to lots more people. For a while we were working our original servers way too hard and the system wasn’t always keeping up with the strain.

Two things have changed: First, we purchased a big pile of fast new servers. Second, we rewrote a lot of the core of PBwiki’s code to scale more effectively. We wrote a lot of new code and also take advantage of a bunch of open source projects, including Pound, Lighttpd, Apache, eAccelerator, MySQL, Memcached, and MogileFS.

happy_idle_servers.png

Shiny, happy, mostly idle servers

We’ve put in a lot of effort to optimizing the load times of wiki pages and in particular speeding up the new editor, since it’s a lot of javascript to load. The ’six times faster’ claim is for the time between when a user requests a wiki page and we’ve finished building it and start sending it to their browser — the ‘time to first byte’. What else does this mean? Well, last month we were averaging 50-100% CPU load on our servers. That means that at midnight in the US, our quietest time of day, we were still using 50% of all available CPU time to serve wiki pages, store edits, and build RSS feeds. During peak time around noon we were starting to back up a bit due to the load — 100% CPU load usually means ‘would go to 150% if we could’. We’re now ranging between 10% and 20% on all of the application servers, which means there’s always CPU time available for the next incoming request. The supporting servers, such as proxy, database, storage and email are even less loaded. This is a very good thing.

At the same time, we’ve got lots more storage headroom. We were running a little low for a while there and now we’ve got at least 6 months of space for new wiki data and attachments, assuming our current growth rates. Combined with our much-improved server monitoring system (thanks to mon, monit, cacti, and some hand-made stuff) we’ve got a great picture of how the service is running and for the first time in a while our servers are as happy as our users.