June 2015 – brionv

One bug down, pass it around

So I fixed a really annoying bug in my video player program where it sometimes just hung waiting on network input.

I thought it was going to be a really hairy multi-threaded concurrency issue, as I was building a blocking i/o interface on top of a non-blocking URL-fetching layer in order to reuse existing libraries.

I scoured my code looking for incorrect ordering, bad locking, or a problem with my semaphores…. Bit by bit the code was instrumented, logged, inspected, cleared, cleaned, and set back in place better than it was before. But the bug remained, untamed — nay untouched — by my onslaught.

At last I found the part of the code the bug was hiding: if the input buffer ran dry and the last network fetch had already completed, the i/o thread would die and the blocking request would time out.

But why was it broken? Because… um… I forgot to call the function that sends a follow-up request for more data.

headdesk

WebM and Ogg energy usage on iOS 9 beta with OGVKit

I’ve been cleaning up some of my old test code for running Ogg media on iOS, adding WebM support and turning it into OGVKit, a (soon-to-be) reusable library that we can use to finally add video and audio playback to our Wikipedia iPhone app.

Of course decoding VP8 or Theora video on the CPU is going to be more expensive in terms of energy usage than decoding H.264 in dedicated silicon… but how much more?

The iOS 9 beta SDK supports enhanced energy monitoring in Xcode 7 beta… let’s try it out! The diagnostic detail screen looks like so:

Whoa! That’s a little overwhelming. What’s actually going on here?

First, what’s going on here

I’ve got my OGVKit demo app playing this video “Curiosity’s Seven Minutes of Terror” found on Wikimedia Commons, on two devices running iOS 9 beta: an iPod Touch (the lowest-end currently sold iDevice) and an iPad Air (one generation behind the highest-end currently sold iDevice).

The iPod Touch is playing a modest 360p WebM transcode, while the iPad Air is playing a higher-resolution 720p WebM transcode with its beefier 64-bit CPU:

First look: the cost of networking

At first, the energy usage looks pretty high:

This however is because in addition to media playback we’re buffering umpty-ump megabytes over HTTPS over wifi — as fast as a 150 Mbps cable connection will allow.

Once the download completes, the CPU usage from SSL decoding goes down, the wifi reduces its power consumption, and our energy usage relatively flattens.

Now what’s the spot-meter look like?

Pretty cool, right!?

See approximate reported energy usage levels for all transcode formats (Ogg Theora and WebM at various resolutions) if you like! Ogg Theora is a little faster to decode but WebM looks significantly better at the bitrates we use.

Ok but how’s that compare to native H.264 playback?

Good question. I’m about to try it and find out.

….

Ok here’s what we got:

The native AVPlayer downloads smaller chunks more slowly, but similarly shows higher CPU and energy usage during download. Once playing only, reported CPU usage dives to a percent or two and the reported energy impact is “Zero”.

Now, I’m not sure I believe “Zero”… ;)

I suppose I’ll have to rig up some kind of ‘run until the battery dies’ test to compare how reasonable this looks for non-trivial playback times… but the ‘Low’ reportage for WebM at reasonable resolutions makes me happier than ‘Very High’ would have!

MediaWiki audio/video support updates

Mirror of mailing list post on wikitech-l and related lists

I’ve been passing the last few days feverishly working on audio/video stuff, cause it’s been driving me nuts that it’s not quite in working shape.

TL;DR: Major fixes in the works for Android, Safari (iOS and Mac), and IE/Edge (Windows). Need testers and patch reviewers.

ogv.js for Safari/IE/Edge

In recent versions of Safari, Internet Explorer, and Microsoft’s upcoming Edge browser, there’s still no default Ogg or WebM support but JavaScript has gotten fast enough to run an Ogg Theora/Vorbis decoder with CPU to spare for drawing and outputting sound in real time.

The ogv.js decoder/player has been one of my fun projects for some time, and I think I’m finally happy with my TimedMediaHandler/MwEmbedPlayer integration patch for the desktop MediaWiki interface.

I’ll want to update it to work with Video.js later, but I’d love to get this version reviewed and deployed in the meantime.

Please head over to https://ogvjs-testing.wmflabs.org/ in Safari 6.1+ or IE 10+ (or ‘Project Spartan’ on Windows 10 preview) and try it out! Particularly interested in cases where it doesn’t work or messes up.

Non-JavaScript fallback for iOS

I’ve found that Safari on iOS supports QuickTime movies with Motion-JPEG video and mu-law PCM audio. JPEG and PCM are, as it happens, old and not so much patented. \o/

As such this should work as a fallback for basic audio and video on older iPhones and iPads that can’t run ogv.js well, or in web views in apps that use Apple’s older web embedding APIs where JavaScript is slow (for example, Chrome for iOS).

However these get really bad compression ratios, so to keep bandwidth down similar to the 360p Ogg and WebM versions I had to reduce quality and resolution significantly. Hold an iPhone at arm’s length and it’s maybe ok, but zoom full-screen on your iPad and you’ll hate the giant blurry pixels!

This should also provide a working basic audio/video experience in our Wikipedia iOS app, until such time as we integrate Ogg or WebM decoding natively into the app.

Note that it seems tricky to bulk-run new transcodes on old files with TimedMediaHandler. I assume there’s a convenient way to do it that I just haven’t found in the extension maint scripts…

In progress: mobile video fixes

Audio has worked on Android for a while — the .ogg files show up in native <audio> elements and Just Work.

But video has been often broken, with TimedMediaHandler’s “popup transforms” reducing most video embeds into a thumbnail and a link to the original file — which might play if WebM (not if Ogg Theora) but it might also be a 1080p original which you don’t want to pull down on 3G! And neither audio nor video has worked on iOS.

This patch adds a simple mobile target for TMH, which fixes the popup transforms to look better and actually work by loading up an embedded-size player with the appropriately playable transcodes (WebM, Ogg, and the MJPEG last-ditch fallback).

ogv.js is used if available and necessary, for instance in iOS Safari when the CPU is fast enough. (Known to work only on 64-bit models.)

Future: codec.js and WebM and OGVKit

For the future, I’m also working on extending ogv.js to support WebM for better quality (especially in high-motion scenes) — once that stabilizes I’ll rename the combined package codec.js. Performance of WebM is not yet good enough to deploy, and some features like seeking are still missing, but breaking out the codec modules means I can develop the codecs in parallel and keep the high-level player logic in common.

Browser infrastructure improvements like SIMD, threading, and more GPU access should continue to make WebM decoding faster in the future as well.

I’d also like to finish up my OGVKit package for iOS, so we can embed a basic audio/video player at full quality into the Wikipedia iOS app. This needs some more cleanup work still.

Phew! Ok that’s about it.

im in ur javascript, decoding ur webm

Been a while since I posted; working on various bits and bobs.

Most exciting at the moment: initial work on pure-JavaScript WebM playback support for ogv.js (soon to become codec.js!)

This is currently very incomplete; video doesn’t come out right and audio’s not hooked up yet, and most notably you only get a few frames before it craps out. :)

Update: Got audio working after some sleep, and playback is more thorough. Yay! Still no seeking yet, and playback may not always reach completion.

I’ve seen a couple other attempts to build WebM decoders in JS using emscripten (eg Route9), but nobody’s gotten them hooked up to a full-blown player with audio yet. With a little more work on the decoder side, I should be able to leverage all of my investment in ogv.js’s player logic.

live preview of current experimental build

How’s it work?

I’m using nestegg for the WebM container parsing; currently this is giving me some impedence mismatch with the first-generation ogv.js code due to different i/o models.

ogv.js’s C-side Ogg container parsing was set up with entirely asynchronous i/o due to the needs of using async XHR to fetch data over the network. Nestegg, however, uses synchronous i/o callbacks in some places, and I really don’t want to jump through all the hoops to do WebM cue seeking without using the library. (Already did that with Ogg and it was horrid!)

I may be able to adapt the ogv.js C-side code to using synchronous i/o callbacks and let them get mapped to async i/o on the JS side, thanks to some fancy features in emscripten — either Asyncify or the newer emterpreter mode for parts of the high-level code while keeping the low-level decoder in pure asm.js. Need to do a little more research… :)

Possibilities

Performance: speed on WebM/VP8 decoding won’t be as good as the Ogg/Theora decoder for now so may not be suitable for mobile devices, but should work fine on many laptop/desktop machines.

VP9: untested so far, but should work… though again, likely to be still slower than VP8.

Threading: currently threading is disabled in the build. There’s work ongoing at Mozilla to support pthreads for emscripten programs, using a new SharedArrayBuffer interface to share memory between Web Worker threads. Might experiment with that later as libvpx has some threading options (may only be useful for VP9 multi-slice decoding for now) but it’ll be a while before most browsers support it. However this might be a good use case for asking Microsoft and Apple to add support. :)

SIMD: there’s also ongoing work on SIMD instruction abstraction in JavaScript, with Mozilla and Microsoft partnering. Will have to test with autovectorization in the compiler enabled and see if it makes any difference to decode speed, but don’t know how complete the support in Firefox and Edge is etc.