Dupe uploads check

We had a post come in this morning on mediawiki-l about an extension for adding a hash-based duplicate file check on upload.

A similar check had been recently added to MediaWiki core, but only on the file description page — it wouldn’t stop your upload while you were in the process. Since all the required backend was there I’ve gone ahead and added it in as a built-in feature for MediaWiki 1.13:

(Note that since we can’t get the file content hash until you upload, there’s still no way to give the warning before you actually upload it. But at least now it lets you cancel at that point instead of having to ask a sysop to come delete your file!)

Wikipedia page size breakdown

Just for kicks, I cleared my cookies & caches and loaded up Wikipedia’s “Frog” article fresh to see what the breakdown of network bandwidth would look like…

645,947 bytes of data content are transferred, not counting any HTTP headers:

72.5% content images
10% JavaScript code
7.5% style sheets
5.5% HTML web page < the important stuff
4.5% UI images

This would take about 90 seconds to download on a 56kbit connection. It’s easy to forget what low-bandwidth feels like for those of us with broadband, but people outside cities may not have good broadband, and mobile devices are often stuck on pretty slow networks too. Compare regular Wikipedia against our mobile gateway on your mobile phone sometime; even a fancy browser like the iPhone’s will feel like molasses trying to load the full site, while loading things up lickety-split from the more minimal mobile gateway.

Fairly simple compression improvements could save 128kb of that:

  • 64k by gzipping JS and CSS files that are currently served uncompressed
  • another 64k through smarter compression of thumbnails (animated GIF optimization, use of JPEG for some PNG thumbs)

That would save approximately 18 seconds of download time for our hypothetical low-bandwidth user.

Details at mw:Wikipedia_data_size_test

More mobile fixlets

Magic quotes strike again!

I’ve disabled magic_quotes_gpc on our mobile transcoder’s PHP configuration, fixing access to articles with apostrophes or double quotes in their names:

Of course, it should be fixed to detect and undo this data corruption on input. At least this misfeature is finally going to die in PHP 6… :D

CentralAuth / SUL is here!

Okay folks, as of a couple hours ago unified login is available opt-in for all Wikimedia accounts!

In addition, we’ve enabled the site-wide global session cookies (which have been in testing for the SSL interface on secure.wikimedia.org for a few weeks). Some people may not be able to successfully get that working across domains (we’ve got reports of Norton blocking the login-cookie-fetching images), but it seems to be working for most people so far. :)

This means that not only will your global, unified account have the same password on say English Wikipedia and Commons, but once you’ve logged in on one you’ll be logged in on the other, without having to log in a second time. Handy!

Note that to do this fully automatically, when you visit a new wiki for the first time it will autocreate a local account for you, linked to your global account. Initially this was spamming the Recent Changes lists with account creation logs, but I’ve now pulled that (they’re still logged in Special:Log, however). (This has been disabled for now, as it’s spamming logs and user lists faster than expected, even through “invisible” links like shared JS and CSS. You’ll still get your shiny local accounts by going through the regular login form, and once you’ve done it once your sessions remain shared.)

Big thanks to Tim Starling who’s done a huge amount of work on CentralAuth in the last couple months, as well as Andrew Garrett who’s helped a lot with the cross-domain cookie logins and global Steward group management.

Mobile gateway search

So it turns out that the search function on Wikipedia’s HawHaw-powered mobile gateway hasn’t been working for a long time, not because it wasn’t implemented, but because it was screen-scraping the search results page.

Some little detail of the results layout changed ages ago, breaking it. Nice! Well, I’ve redone it to use the MediaWiki web service API which should be a little more stable.

Search works again, yay!

Even if the correct search result is fifth in the output *cough* :)

Hey, we’re workin’ on it. ;)

More CentralAuth comin’ Tuesday

Hey, just to give y’all a heads-up… after a couple months of good testing w/ the sysops & power users, we’re going to widen the CentralAuth rollout to allow everybody on Wikimedia sites to opt-in to the system.

We’re going to keep automatic migration off for now to keep the volume down, as we may want to roll out more helper tools in response to new issues people might have.