How I spent my election night

This is shaping up as a busy week at Wikimedia HQ… We’re starting our year-end fundraiser, our remote staff are flying into town for an all-staff meeting, and to top it off there was this election thingy going on…

Over the last few weeks we’ve all been scrambling to get things ready. The fundraiser landing pages are beautiful, our donation tracking system seems to actually work (thanks David!), and the site notice banner doesn’t have any scrolling marquees… ;)

Thanks to the tireless work of Tomasz and Trevor, we now have a workable system for previewing notice banner updates and scheduling them before they go live — which should make things much smoother as the designs get updated over the course of the fundraiser season.

We started live-testing the notice Monday on test.wikipedia.org:

By Tuesday morning, Rand and the rest of the fundraiser team were rushing to fix up the donation pages in response to review and broken-link testing, and we started deploying notices live — first to Wikibooks, then Wikipedia in Japanese, Polish, French, and German.

After a few hours of testing and tweaking, we stuck the notice on full on all Wikipedias… And saw a prompt decline in site traffic as several of our proxy cache servers threw themselves into some kind of overload failure.

This being a) the start of our fundraiser and b) the night of the US Presidential election, we thought we’d better stay in and fix it instead of going out for a tech department dinner…

I canceled our reservation and ordered some Indian delivery while Tim and Mark sorted out the servers, and we kept a sharp eye on things as the evening continued…

The overload may have been triggered by extra hits to backend web servers for the images in the notice; to be on the safe side we moved them over to the upload/media servers before redeploying.

Around the time the networks started calling the election for Obama, we were ready to put the notices back in. We were seeing some load spikes on backend web servers, but these seemed to be from legitimate page rendering, most likely related to the heavy editing traffic for election results.

As McCain conceded the election, we started filtering in the fundraising notice for about 1 in ten hits, making sure it wasn’t breaking anything.

By the middle of Obama’s victory speech, we’d ramped it up to 100% of hits, and started to see the donations rolling in.

We’re seeing definite positive response — people are certainly turning out to support Wikipedia! We’re also getting feedback on the layout of the notice itself, and are making some tweaks in response. For starters we’re changing the “Hide” link to a more accurate “Collapse”; like last year’s the link doesn’t completely hide the notice, just collapses it to a smaller version.

Thanks everybody for your support (and I hope you voted, too, if you were eligible!) — we’ll try to keep things smooth on our end and keep the fundraiser classy.