Between our high traffic, our wacky insane number of edits, new software features, and the ever-growing amount of stuff on Wikipedia and friends, the demand on our servers is always going up.
Our stack has several layers, from the MySQL database backend to the geographically distributed Squid caches, but the heavy lifting of all that wiki page formatting and editing logic is handled in (at last count) 156 Apache/PHP servers running MediaWiki.
We last did a major expansion of these application servers in mid-2007, which ended up holding us a lot longer than we’d originally anticipated. In the last couple of weeks we’ve finally started hitting up against some capacity limits at peak times — especially Mondays, around the afternoon in North America and evening in Europe — making everything horribly slow.
We’ve recently retired some of our oldest web servers to free up space for a full rack of newer, faster ones; an “energy-efficient” variant of the Dell PowerEdge 1950, packing 8 cores of awesome at 2.5 GHz each. (We’ll follow up on the fate of the retired boxes in another post…)
With just 9 of the 36 new boxes installed so far, we’ve already seen a visible decrease in page service times today:
According to Mark’s estimates tweaking the load balancer, these 9 boxes alone are serving as much traffic as our 41 oldest boxes still in service — with lower electricity usage, making them cheaper to run. Moore’s Law wins for you again!
The remaining 27 new boxes will be going into service over the next couple days, improving performance further.
Wow. That makes me happy. The stuff I work for goes to good use.
Awesome sauce.
-R
Looks good glad to hear your upgrading the servers every so often
Kewl!
You should put this on the WMF blog as well.
And I was wondering with some of the slowdowns if the IWF was doing crap web filtering again …
Great news!
I wonder why Dell cannot just donate some of these servers. it isn’t going to hurt them.