Google Transit yay!

A few months ago I whined about the Google Maps transit planner not working very well.

Well somewhere since I last looked, they fixed it!

Transit directions now include San Francisco MUNI bus and train routes and walking to/from stations, so you can actually put in start and end points and get something useful! The alternate route selection is a little different from the driving directions (you get a short list of a few options, rather than being able to click and drag waypoints to whatever route you like), but still quite useful; it comes up with pretty close facsimiles to the three alternate commute routes I use in reality.

Goodbye, 511.org!

Now if they can just integrate the transit lookups into the iPhone Google Maps widget… d’oh!

German FlaggedRevs tested for 10 minutes

Ok, so we finally got the FlaggedRevs for German Wikipedia config set up… then turned it off after a few minutes.

We did, alas, encounter a few problems, which didn’t come up as much in earlier testing, but came up *hard* in a few minutes around 3am at Wikipedia. :)

Floating UI boxes and floating infoboxes don’t mix well.

The nice small versioning marker is really nice, but that’s way too disruptive, and we’ll need to get it worked out one way or another.

Second, some of the reporting pages weren’t working, in part due to some last-minute tweaks to the DB layout to make it easier to deploy. (This should be fixed now.)

Third, the “redirected from” subtitles are being broken, which’ll disrupt some general editing functionality in an unpleasant way. An example on de.labs test wiki.

Once the UI bits are fixed up, we’ll give it another test run… und FlaggedRevs kommt wieder!

Top 10 Wikimedia DB errors

I did a quick look last night through our database error logs for the last week or so, breaking them down by function and error type. Here’s the top ten function-err loci:

Hits Function errno Error
620 Article::updateCategoryCounts 1213 Deadlock found when trying to get lock; Try restarting transaction
240 Article::insertOn 1062 Duplicate entry ‘N-XXX’ for key 2
41 Article::doDeleteArticle 1213 Deadlock found when trying to get lock; Try restarting transaction
26 LinksUpdate::incrTableUpdate 1213 Deadlock found when trying to get lock; Try restarting transaction
19 TitleKey::prefixSearch 1030 Got error 28 from table handler
9 Title::invalidateCache 1213 Deadlock found when trying to get lock; Try restarting transaction
9 2013 Lost connection to MySQL server during query
8 User::saveSettings 1205 Lock wait timeout exceeded; Try restarting transaction
8 TitleKey::prefixSearch 2003 Can’t connect to MySQL server on ‘XXX’
7 Job::pop 1213 Deadlock found when trying to get lock; Try restarting transaction

A large chunk of our DB errors are from conflicting transactions; the number one spot is currently taken up by updates to category counts, which is often part of an expensive page deletion transaction.

We’re often pretty lazy about rerunning database transactions when they’re rolled back, throwing an error and making the end-user resubmit the change. This is kind of lame, but at least the transaction rollback theoretically keeps the database consistent.

The number two spot seems to be for conflicting page creations — possibly due to automatic resubmissions after a slow save operation.

There’s a few “disk full” errors, which were probably due to a transitory error on one DB box.

Diff bug fixed, hopefully

For a long time we’ve had intermittent problems with diffs displaying incorrectly, with lines on the left side mysteriously repeated:

Reports skyrocketed the other day, when the wikidiff2 extension (our C++ reimplementation of MediaWiki’s diff algorithm, about a billion times faster than the PHP one) was upgraded to match upgrades of PHP on our older, Fedora Core-based servers.

I added in some logging hacks to try to track it down, but didn’t get a lot of data points until I tried the simple expedient of running every diff twice — if the results don’t match, log the error.

With a few hundred instances logged, it became clear that the problem was limited to servers running Fedora 4; even-older Fedora 3 boxes were unaffected, as were all our newer Ubuntu boxes. Mysterious problems caused by C++ run-time library mismatches between different Linux releases are not at all uncommon; it looked like we’d installed an FC3 binary on all the machines, and it was intermittently failing on FC4.

I recompiled the extension, this time with separate builds on FC3 and FC4, and haven’t seen any bad diffs come through my log in the last half hour… so far so good! :)

So what’s in the job queue anyway?

In en.wikipedia.org’s job queue at the moment, breakdown by job type…

job_cmd count(*)
htmlCacheUpdate 31,147
refreshLinks 10,106,739
renameUser 119

Note that the current system allows for duplicate entries to get put in the queue; the dupes are removed as the first one in the stack gets run. This makes the raw number of refreshLinks entries much higher than it “really” is — [[Wikipedia:Talk:Union Station (Louisville)|Talk:Union Station (Louisville)]] is listed 9 times, presumably once for each template edit that triggered an “update me!” job.

Update: Figured out why the queues were growing so big last few days — system clock was 7 seconds slow on the database master. This made the replication lag detection misread a 7-second minimum lag on every slave. The job queue batch runners were all sitting waiting for the lag to resolve. :)

Resynced the clock (presumably drifted during the period when some IPs were broken), things are moving again.

Visual Voicemail fixed

One of the oh-so-cute features of the iPhone is visual voicemail, the “duh” feature of showing you an actual on-screen list of your voicemails instead of making you sit through voice prompts.

Bad: My iPhone mysteriously reverted to the classic “press 7 to delete” system when I changed rate plans a couple weeks ago… with voice mail disabled altogether so callers couldn’t leave messages until I noticed it and set up a new password.

A little Googling indicates this is a fairly common mix-up, and the only way to restore visual voicemail is to call AT&T tech support and have them fiddle with your account settings.

Good: AT&T tech support was able to fix the account settings so it works again… after a half hour on hold… :)

WTF: The AT&T tech swore that visual voicemail doesn’t work if you have a WiFi connection active. He had me disable WiFi while initially testing it, then when I asked him about it he told me outright that Visual Voicemail only works on the EDGE network and therefore you must turn off WiFi to check your voicemail.

This is demonstrably false; just to confirm I hadn’t been crazy for the couple of months my voicemail was working just fine, I turned WiFi back on, left myself a voicemail, and retrieved it just fine in all its visual glory.

It’s entirely possible that the voicemails still download over EDGE, but having the WiFi up doesn’t seem to interfere at all.

Now if they can just add a feature to route phone calls over WiFi, I could actually get calls through from my flat. ;)

Suggestion search drop-down

Another in today’s series of fun feature enablings…

The search boxes on Wikimedia wikis now have an AJAX-powered search suggestion drop-down. This calls our JSON OpenSearch suggestion interface, which has been used for some time by Firefox’s search box and Mac OS X 10.5’s Dictionary application, but is now built-in for your viewing pleasure.

(In MediaWiki 1.13 development trunk, turn on $wgEnableMWSuggest to experience this yourself!)

A similar AJAX-powered search feature has been in MediaWiki for some time, but the user interface for it took over the whole article area, which was a bit distracting, and we never used it ourselves.

Robert Stojnic, the tireless coder who’s put a huge amount of effort into fixing up our Lucene-based search engine over the last months, patched up the front-end to fit more naturally into the existing forms.

The built-in search for suggestions is currently a simple prefix match, so it’ll help you complete words and names, but isn’t smart enough to fill out from a last name or skip “the” etc. Robert’s got a new backend in the works, which will add all those smarts when we’re ready to upgrade the search systems with the new software and a bit beefier hardware.

Prefix matches are a heck of a lot better than nothing, though, and as long as it’s not causing undue server load we’ll keep it on until the new backend’s ready.

(If you don’t like the suggestions widget, you can disable them by checking “Disable AJAX suggestions” in the “Search” tab at Special:Preferences.)