For a long time we’ve had intermittent problems with diffs displaying incorrectly, with lines on the left side mysteriously repeated:
Reports skyrocketed the other day, when the wikidiff2 extension (our C++ reimplementation of MediaWiki’s diff algorithm, about a billion times faster than the PHP one) was upgraded to match upgrades of PHP on our older, Fedora Core-based servers.
I added in some logging hacks to try to track it down, but didn’t get a lot of data points until I tried the simple expedient of running every diff twice — if the results don’t match, log the error.
With a few hundred instances logged, it became clear that the problem was limited to servers running Fedora 4; even-older Fedora 3 boxes were unaffected, as were all our newer Ubuntu boxes. Mysterious problems caused by C++ run-time library mismatches between different Linux releases are not at all uncommon; it looked like we’d installed an FC3 binary on all the machines, and it was intermittently failing on FC4.
I recompiled the extension, this time with separate builds on FC3 and FC4, and haven’t seen any bad diffs come through my log in the last half hour… so far so good! :)