ogv.js MediaWiki integration updates

Over the last few weekends I’ve continued to poke at ogv.js, both the core library and the experimental MediaWiki integration. It’s getting pretty close to merge-ready!

Recent improvements to ogv.js player (gerrit changeset):

  • Audio no longer super-choppy in background tabs
  • ‘ended’ is no longer unreasonably delayed
  • various code cleanup
  • ogvjs-version.js with build timestamp available for use as a cache-buster helper

Fixes to the MediaWiki TimedMediaHandler desktop player integration (gerrit changeset):

  • Post-playback behavior is now the same as when using native playback
  • Various code cleanup

Fixes to the MediaWiki MobileFrontend mobile player integration (gerrit changeset):

  • Autoplay now working with native playback in Chrome and Firefox
  • Updated to work with current MobileFrontend (internal API changes)
  • Mobile media overlay now directly inherits from the MobileFrontend photo overlay class instead of duplicating it
  • Slow-CPU check is now applied on mobile player — this gets ogv.js video at 160p working on an old iPhone 4S running iOS 7! Fast A7-based iPhones/iPads still get 360p.

While we’re at it, Microsoft is opening up a public ‘suggestion box’ for Internet Explorer — folks might want to put in their votes for native Ogg Vorbis/Theora and WebM playback.

Testing ogv.js in MediaWiki

After many weekends of development on the ogv.js JavaScript Ogg Vorbis/Theora player I’ve started work on embedding it as a player into MediaWiki’s TimedMediaHandler extension.

The JavaScript version is functional in Safari and IE 10/11, though there’s some work yet to be done… See a live wiki at ogvjs-testing.wmflabs.org or the in-progress patch set.

Screen Shot 2014-07-13 at 8.43.34 PM


Meanwhile, stay tuned during the week for some demos of the soon-to-be-majorly-updated Wikipedia iOS app!

OgvKit: native Ogg Vorbis/Theora playing on iOS

In addition to my in-browser ogv.js media player, I’ve got an OgvKit framework in progress for native iPhone and iPad apps, which I hope to integrate into Wikipedia’s new iOS app somewhere down the line. I took a little Independence Day holiday time and made a bunch of improvements from where I last left it a few months ago:

(If the video is sideways, sorry — it’s still “processing” as of this writing.)

  • Color conversion is OpenGL ES-accelerated, cutting CPU usage in half when playing video.
  • Audio output actually works, more or less in sync.
  • Framework now packaged as a Cocoa Touch Static Library project

Shiny new A7-based 64-bit devices play 480p and even 720p happily (iPhone 5S, iPad Air) but still struggle with some 1080p originals. My oldest devices like an iPhone 3Gs and iPod Touch 4th-gen can’t go higher than 160p or so at present — still significantly faster than the JavaScript ogv.js version which can’t handle video on those devices at all.

Future work for another weekend:

  • It turns out the xiph.org git mirrors of theora etc are not being maintained; will switch fetching of library source to SVN.
  • Improved controls, seeking, fullscreen
  • Cache downloaded data on disk instead of RAM!
  • Move more of the player code from the demo into the library so it can be reused…
  • Test standalone packaging of the library so it can be dropped in to other projects easily
  • Try to get assembly in libtheora to build, and/or replace the ARM assembly code with C SIMD intrinsics
  • Find and test on an armv7s device (iPhone 5, 5C or iPad 4th-generation)

iPhone size speculation

So there’s wide speculation out there that a larger-screened iPhone is coming to compete with the circa-5-inch generation of flagship Android phones.

But how would Apple actually engineer such a thing, and what would it mean for web and app developers?

There are a few possibilities to my mind, with various trade offs.

First, note that the iPhone 5 family has a 4″ 640×1136 screen, about 326dpi and using a 2x scale between UI points (aka “CSS pixels”). Software for iPhone had long assumed a 320×480 1x or 640×960 2x display, and when the new screen size was introduced, old apps were accommodated by showing them with black bars to simulate the older screen size.

The simplest solution would be to scale up the iPhone 5c/s design and screen by the same ratio as the iPad mini and iPad Air — this gives a 4.8 inch “iPhone 6″ that would have the same pixel density as the full-size Retina iPad. Developers would see the 4″ and 4.8″ devices as equivalent, with no changes needed to code or graphics. But, it might not provide the feeling of additional screen space because on screen elements would become larger to fit. The resolution, while still a respectable 263dpi or so would also fall noticeably short of the current crop of 1080p Android phones at 440+dpi.

Another possibility is to stick to the 326dpi density and change the screen’s pixel dimensions, say to 1280×720 at 4.5” or maybe a little higher. This would give more onscreen space, requiring app developers to ensure they handle the different dimensions. Older apps without AutoLayout might be handled by a black screen border like iPhone apps running on an iPad, or they might scale up and be a little blurry.

Some might scoff at such a change, but iOS SDK updates have made it easier to handle varying screen sizes, from iOS 6’s introduction of AutoLayout to iOS 8’s storyboard unification and mysterious “variable sized iPhone simulator”… The idea that Apple might spring a new form factor at us in fall 2014 is not any crazier than when they sprung the 4” iPhone 5 on us…

Still another question is whether Apple will follow the dpi race that they started with the iPhone 4’s “retina” screen… Would they make a 1080p phone in the 4.5-5″ range using a 3x display scale to match Android? There’s nothing in the iOS SDK that hints that way to me, but it’s plausible technically.

A 3x density scale might handle back-compatibility by rendering to a 2x frame buffer and transparently scaling up at some cost of blurriness, while newer apps use @3x artwork and render natively at proper scale.

This remains to be seen…

Safely embedded JavaScript modules for MediaWiki?

At the Zürich Hackathon I’ve been poking a number of things — more notes to come later — but one fun one is that I’ve continued work on one of my older experiments, getting a demo running on Wikimedia Labs and starting to enhance it for MediaWiki’s relatively new ContentHandler system to create custom structured ‘pages’.

Screen Shot 2014-05-11 at 1.20.04 PM

(TL;DR if you don’t want to click through to the docs on the extension: an isolated iframe is used to sandbox script code from the wiki’s own web environment.)

Currently the ‘JSApplet’ namespace content handler just takes straight JavaScript source, but I’m planning to make it a structured bundle which contains:

  • an HTML scaffold
  • a CSS chunk
  • a JavaScript chunk
  • references to other scripts to include
  • references to SVG or raster image files to include
  • print or non-JS fallback content

Then, a custom view/edit handler for the content type can provide an interface to edit those bits and run an immediate preview — kind of like an embedded JSFiddle.

ContentHandler also allows for a custom transclusion mode — so scripts could perhaps be invoked like ‘{{JSApplet:Mandelbrot}}” instead of having to use an XML-like tag extension manually. Not sure if that’s the best plan, but it’s a thought. 😀

I’m also thinking about how to make this work on mobile — in theory these things should be able to work with iframes in embedded web views, but it may require adding special support in the client.

For times when the script can’t be executed, some sort of static fallback content should be shown — still thinking about best ways to support that cleanly and make it something that people will do by default. Hmm, ideas ideas. :)


Internet Explorer on Windows Phone 8.1 is finally debuggable

Fun fact — Visual Studio 2013 Express for Windows allows connecting the JavaScript debugger to a Windows Phone 8.1’s device’s mobile Internet Explorer. Not quite as convenient as Safari and Chrome’s support for iOS and Android web debugging, but a hell of an improvement from “throw some alert()s in there to see what’s going on”.

ogv.js now in WebGL and Flash flavors

Two major updates to ogv.js Theora/Vorbis video/audio player in the last few weekends: an all-Flash decoder for older IE versions, and WebGL and Stage3D GPU acceleration for color conversion and drawing.

Try the demo and select ‘JS + WebGL‘, ‘Flash’, or even ‘Flash + GPU‘ from the drop-down! (You can also now try playback in the native video element or the old Cortado Java applet for comparison, though Cortado requires adding security exceptions if your browser works with Java at all these days. :P)

Flash / Crossbridge

The JS code output by emscripten requires some modern features like typed arrays, which aren’t available on IE 9 and older… but similar features exist in Flash’s ActionScript 3 virtual machine, and reasonably recent Flash plugins are often available on older IE installations.

The Flash target builds the core C codec libraries and the C side of the OgvJs codec wrapper using Crossbridge, the open-source version of Adobe’s FlasCC cross-compiler. This is similar in principle to the emscripten compiler used for the JavaScript target, but outputs ActionScript 3 bytecode instead of JavaScript source.

I then ported the JavaScript code for the codec interface and the player logic to ActionScript, wrapped it all up into a SWF using the Apache Flex mxmlc compiler, and wrapped the Flash player instantiation and communication logic in a JavaScript class with the same interface as the JS player.

A few interesting notes:

  • Crossbridge compiler runs much slower than emscripten does, perhaps due to JVM startup costs in some backend tool. Running the configure scripts on the libraries is painfully slowwwwwwwwww! Luckily once they’re built they don’t have to be rebuilt often.
  • There are only Mac and Windows builds of Crossbridge available; it may or may not be possible to build on Linux from source. :( I’ve only tested on Mac so far.
  • Flash decoder performance is somewhere on par with Safari and usually a bit better than Internet Explorer’s JS.
  • YCbCr to RGB conversion in ActionScript was initially about 10x slower than the JavaScript version. I mostly reduced this gap by moving the code from ActionScript to C and building it with the Crossbridge compiler… I’m not sure if something’s building with the wrong optimization settings or if the Crossbridge compiler just emits much cleaner bytecode for the same code. (And it was literally the same code — except for the function and variable declarations the AS and C code were literally identical!)
  • ActionScript 3 is basically JavaScript extended with static type annotations and a class-based object system with Java-like inheritance and member visibility features. A lot of the conversion of JS code consisted only of adding type annotations or switching HTML/JS-style APIs for Flash/AS ones.
  • JS’s typed array & ArrayBuffer system doesn’t quite map to the Flash interfaces. There’s a ByteArray which is kind of like a Uint8Array plus a data stream interface, with methods to read/write values of other types into the current position and advance it. There are also Vector types, which have an interface more like the traditional untyped Array but can only contain items of the given type and are more efficiently packed in memory.

GPU acceleration: WebGL

WebGL is a JavaScript API related to OpenGL ES 2, exposing some access to GPU programming and accelerated drawing of triangles onto a canvas.

Safari unfortunately doesn’t enable WebGL by default; it can be enabled as a developer option on Mac OS X but requires a device jailbreak on iOS.

However for IE 11, and the general fun of it, I suspected adding GPU acceleration might make sense! YCbCr to RGB conversion and drawing bytes to the canvas with putImageData() are both expensive, especially in IE. The GPU can perform the two operations together, and more importantly can massively parallelize the colorspace conversion.

I snagged the fragment shader from the Broadway H.264 player project’s WebGL accelerated canvas, and with a few tutorials, a little documentation, and a lot of trial and error I got accelerated drawing working in Firefox, Chrome, and Safari with WebGL manually enabled.

Then came time to run it on IE 11… unfortunately it turns out that single-channel luminance or alpha textures aren’t supported on IE 11’s WebGL — you must upload all textures as RGB or RGBA.

Copying data from 1-byte-per-pixel arrays to a 3- or 4-byte-per-pixel array and then uploading that turned out to be horribly slow, especially as IE’s typed array ‘set’ method and copy constructor seem to be hideously slow. It was clear this was not going to work.

I devised a method of uploading the 1-byte-per-pixel arrays as pseudo-RGBA textures of 1/4 their actual width, then unpacking the subpixels from the channels in the fragment shader.

The unpacking is done by adding two more textures: “stripe” textures at the luma and chroma resolutions, which for each pixel have a 100% brightness in the channel for the matching packed subpixel. For each output pixel, we sample both the packed Y, Cb, or Cr texture and the matching-size stripe texture, then multiple the vectors and sum the components to fold down only the relevant channel into a scalar value.

It feels kinda icky, but it seems to run just as fast at least on my good hardware, and works in IE 11 as well as the other browsers.

On Firefox with asm.js-optimized Theora decoding and WebGL-optimized drawing, I can actually watch 720p and some 1080p videos at full speed. Nice! (Of course, Firefox can already use its native decoder which is faster still…)

A few gotchas with WebGL:

  • There’s a lot of boilerplate you have to do to get anything running; spend lots of time reading those tutorials that start with a 2d triangle or a rectangle until it makes sense!
  • Once you’ve used a canvas element for 2d rendering, you can’t switch it to a WebGL canvas. It’ll just fail silently…
  • Creating new textures is a lot slower than uploading new data to an existing texture of the same size.
  • Error checking can be expensive because it halts the GPU pipeline; this significantly slows down the rendering in Chrome. Turn it off once code is working…

GPU acceleration: Flash / Stage3D

Once I had things working so nicely with WebGL, the Flash version started to feel left out — couldn’t it get some GPU love too?

Well luckily, Flash has an OpenGL ES-like API as well: Stage3D.

Unluckily, Stage3D and WebGL are gratuitously different in API signatures. Meh, I can work with that.

Really unluckily, Stage3D doesn’t include a runtime shader compiler.

You’re apparently expected to write shaders in a low-level assembly language, AGAL… and manually grab an “AGALMiniAssembler” class and stick it in your code to compile that into AGAL bytecode. What?

Luckily there’s also a glsl2agal converter, so I was able to avoid rewriting the shaders from the WebGL version in AGAL manually. Yay! This required some additional manual hoop-jumping to make sure variables mapped to the right registers, but the glsl2agal compiler makes the mappings available in its output so that wasn’t too bad.

Some gotchas with Stage3D:

  • The AS3 documentation pages for Stage3D classes don’t show all the class members in Firefox. No, really, I had to read those pages in Chrome. WTF?
  • Texture dimensions must be powers of 2, so I have to copy rows of bytes from the Crossbridge C heap into a temporary array of the right size before uploading to the GPU. Luckily copying byte arrays is much faster in Flash than in IE’s JS!
  • As with the 2d BitmapData interface, Flash prefers BGR over RGB. Just had to flip the order of bytes in the stripe texture generation.
  • Certain classes of errors will crash Flash and IE together. Nice!
  • glsl2agal compiler forced my texture sampling to linear interpolation with wrapping; I had to do a string replace on the generated AGAL source to set it to nearest neighbor & clamped.
  • There doesn’t appear to be a simple way to fix the Stage3D backing buffer size and scale it up to fit the window, at least in the modes I’m using it. I’m instead handling resizing by setting the backing buffer to the size of the stage, and just letting the texture render larger. This unfortunately uses nearest-neighbor for the scaling because I had to disable linear sampling to do the channel-packing trick.
  • On my old Atom-based tablet, Stage3D drawing is actually slower than software conversion and rendering. D’oh! Other machines seem faster, and about on par with WebGL.

I’ll do a little more performance tweaking, but it’s starting to look like it’s time to clean it up and try integrating with our media playback tools on MediaWiki… Wheeeee!


ogv.js now with synchronized sound

My ogv.js JavaScript Theora video player project now has synchronized audio and video, which makes it possible to watch videos of people talking without going mad. 😀

Screen Shot 2014-03-06 at 4.57.30 AM

Try it out!

This works both with Web Audio in Firefox, Chrome, and Safari and with the Flash audio shim for IE; basically we have to keep track of the audio playback position and match up decoding frames with that. Took a little poking in the ActionScript code but it’s now working!

This brings us much closer to being able to integrate ogv.js as a fallback video player for Wikimedia on IE 10/11 and Safari 6/7. Thanks to the guys hanging out in #xiph channel who encouraged me to keep poking at this! 😀

Additionally there’s now an override selector for the video size, so you can try decoding larger than 360p versions, or switch a slow machine down to the little 160p versions.

I’ve also started investigating an all-Flash version using Adobe’s Crossbridge, which if it works would be a suitable replacement for the Cortado Java applet on old browsers (think of all those IE 6/7/8/9 systems out there!). I seem to be able to build the ogg libraries but haven’t gone beyond that yet… will be interesting to poke at.