ogv.js soft launch on Wikipedia and Wikimedia Commons

Soft launch of ogv.js on Wikipedia and Wikimedia Commons has begun! This initial deployment covers the desktop view only, so iPhones and iPads won’t get the media player yet in mobile view.

ogv.js provides a JavaScript compatibility shim for Ogg audio and video playback in Safari 6.1 and higher, IE 10/11, and Microsoft Edge browsers, which gets Wikipedia’s media files working in those browsers. (Due to patent licensing concerns, we don’t provide files in the common MP3 or MP4 H.264/AAC formats, and this has made it difficult to use media files reliably across browsers as Apple and Microsoft have not adopted the free Ogg or WebM formats.)

See list of pending fixes for additional improvements that should go out next week, after which I’ll make wider announcements.

Here, have some samples! In Firefox, Chrome, or Opera these will “just work” with native WebM playback, while in Safari/IE/Edge they will “just work” with JavaScript Ogg playback:

Curiosity’s Seven Minutes of Terror

 

Caminandes – Gran Dillama

 

Using Web Worker threading in ogv.js for smoother playback

I’ve been cleaning up MediaWiki’s “TimedMediaHandler” extension in preparation for merging integration with my ogv.js JavaScript media player for Safari and IE/Edge when no native WebM or Ogg playback is available. It’s coming along well, but one thing continued to worry me about performance: when things worked it was great, but if the video decode was too slow, it could make your browser very sluggish.

In addition to simply selecting too high a resolution for your CPU, this can strike if you accidentally open a debug console during playback (Safari, IE) or more seriously if you’re on an obscure platform that doesn’t have a JavaScript JIT compiler… or using an app’s embedded browser on your iPhone.

Or simply if the integration code’s automatic benchmark overestimated your browser speed, running too much decoding just made everything crap.

Luckily, the HTML5 web platform has a solution — Web Workers.

Workers provide a limited ability to do multithreading in JavaScript, which means the decoder thread can take as long as it wants — the main browser thread can keep responding to user input in the meantime.

The limitation is that scripts running in a Worker have no direct access to your code or data running in the web page’s main thread — you can communicate only by sending messages with ‘raw’ data types. Folks working with lots of DOM browser nodes thus can’t get much benefit, but for buffer-driven compute tasks like media decoding it’s perfect!

Threading comms overhead

My first attempt was to take the existing decoder class (an emscripten module containing the Ogg demuxer, Theora video decoder, and Vorbis audio decoder, etc) and run it in the worker thread, with a proxy object sending data and updated object properties back and forth.

This required a little refactoring to make the decoder interfaces asynchronous, taking callbacks instead of returning results immediately.

It worked pretty well, but there was a lot of overhead due to the demuxer requiring frequent back-and-forth calls — after every processing churn, we had to wait for the demuxer to return its updated status to us on the main thread.

This only took a fraction of a millisecond each time, but a bunch of those add up when your per-frame budget is 1/30 (or even 1/60) second!

 

I had been intending a bigger refactor of the code anyway to use separate emscripten modules for the demuxer and audio/video decoders — this means you don’t have to load code you won’t need, like the Opus audio decoder when you’re only playing Vorbis files.

It also means I could change the coupling, keeping the demuxer on the main thread and moving just the audio/video decoders to workers.

This gives me full speed going back-and-forth on the demuxer, while the decoders can switch to a more “streaming” behavior, sending packets down to be decoded and then displaying the frames or queueing the audio whenever it comes back, without having to wait on it for the next processing iteration.

The result is pretty awesome — in particular on older Windows machines, IE 11 has to use the Flash plugin to do audio and I was previously seeing a lot of “stuttery” behavior when the video decode blocked the Flash audio queueing or vice versa… now it’s much smoother.

The main bug left in the worker mode is that my audio/video sync handling code doesn’t properly handle the case where video decoding is consistently too slow — when we were on the main thread, this caused the audio to halt due to the main thread being blocked; now the audio just keeps on going and the video keeps playing as fast as it can and never catches up. ūüôā

However this should be easy to fix, and having it be wrong but NOT FREEZING YOUR BROWSER is an improvement over having sync but FREEZING YOUR BROWSER. ūüôā

WebGL performance tricks on MS IE and Edge

One of my pet projects is ogv.js, a video/audio decoder and player in JavaScript powered by codec libraries ported from C with Mozilla’s emscripten transpiler. I’m getting pretty close to a 1.0 release and deploying it to Wikimedia Commons to provide plugin-free Ogg (and experimentally WebM) playback on Apple’s Safari and Microsoft’s Internet Explorer and Edge browsers (the only major browsers lacking built-in WebM video support).

In cleaning it up for release, I’ve noticed some performance regressions on IE and Edge due to cleaning out old code I thought was no longer needed.

In particular, I found that drawing and YUV-RGB colorspace conversion using WebGL, which works fantastically fast in Safari, Chrome, and Firefox, was about as slow as on-CPU JavaScript color conversion in IE 11 and Edge — luckily I had a hack in store that works around the bottleneck.

It turns out that uploading single-channel textures as LUMINANCE or ALPHA formats is vveerryy ssllooww in IE 11 update 1 and Edge, compared to uploading the exact same data blob as an RGBA texture…

ogvjs-win10-faster

As it turns out, I had had some older code to stuff things into RGBA textures and then unpack them in a shader for IE 10 and the original release of IE 11, which did not support LUMINANCE or ALPHA texture uploads! I had removed this code to simplify my WebGL code paths since LUMINANCE got added in IE 11 update 1, but hadn’t originally noticed the performance regression.

Unfortunately this adds a user-agent sniff to my ogv.js code… I prefer to use the LUMINANCE textures directly on other browsers where they perform well, because the textures can be scaled more cleanly in the case of source files with non-square pixels.

Wikimedia video community editing tools & infrastructure status

There were a fair number of folks interested in video chatting at Wikimania! A few quick updates:

An experimental ‘Schnittserver’ (‘Clip server’) project has been in the works for a while with some funding from ze Germans; currently sitting at http://wikimedia.meltvideo.com/ (uses OAuth, has a temporary SSL cert, UI is very primitive!) It is currently usable already for converting MP4 etc source footage to WebM!

The Schnittserver can also do server-side rendering of projects using the ‘melt’ format such as those created with Kdenlive and Shotcut — this allows uploading your original footage (usually in some sort of MP4/H.264 flavor) and sharing the editing project via WebM proxy clips, without generational loss on the final rendering.

Once rendered, your final WebM output can be published up to Commons.

I would love to see some more support for this project, including adding a better web front-end for managing projects/clips and even editing…

Mozilla has an in-browser media editor thing called Popcorn.js; they’re unfortunately reducing investment in the project, but there’s some talk among people working on it and on our end that Wikimedia might be interested in helping adapt it to work with the Schnittserver or some future replacement for it.

Unfortunately I missed the session with the person working on Popcorn.js, will have to catch up later on it!

I’m very close to what I consider a 1.0 release of ogv.js, my JavaScript shim to play Ogg (and experimentally WebM) video and audio in Safari and MS IE/Edge without plugins.

Recently fixed some major sound sync bugs on slower devices, and am finishing up controls which will be used in the mobile view (when not using the full TimedMediaHandler / MwEmbedPlayer interface which we still have on the desktop).

Demo of playback at https://brionv.com/misc/ogv.js/demo2/

A slightly older version of ogv.js is also running on https://ogvjs-testing.wmflabs.org/ with integration into TimedMediaHandler; I’ll update those patches with my 1.0 release next week or so.

Infrastructure issues:

I had a talk with Faidon about video requirements on the low-level infrastructure layer; there are some things we need to work on before we really push video:

– seeking/streaming a file with Range subsets causes requests to bypass the Varnish cache layer, potentially causing huge performance problems if there’s a usage spike!

– very large files can’t be sharded cleanly over multiple servers, which makes for further performance bottlenecks on popular files again

– VERY large files (>4G or so) can’t be stored at all; which is a problem for high-quality uploads of things like long Wikimania talks!

For derivative transcodes, we can bypass some of these problems by chunking the output into multiple files of limited length and rigging up ‘gapless playback’, as can be done for HLS or MPEG-DASH-style live streaming. I’m pretty sure I can work out how to do this in the ogv.js player (for Safari and IE) as well as in the native <video> element playback for Chrome and Firefox via Media Source Extensions. Assuming it works with the standard DASH profile for WebM, this is something we can easily make work on Android as well using Google’s ExoPlayer.

DASH playback will also make it easier to use adaptive source switching to handle limited bandwidth or CPU resources.

However we still need to be able to deal with source files which may be potentially quite large…

List and phab projects!

As a reminder there’s a wikivideo-l list: https://lists.wikimedia.org/mailman/listinfo/wikivideo-l

and a Wikimedia-Video project tag in phabricator: https://phabricator.wikimedia.org/tag/wikimedia-video/

Folks who are interested in pushing further work on video, please feel free to join up. There’s a lot of potential awesomeness!

WebM and Ogg energy usage on iOS 9 beta with OGVKit

I’ve been cleaning up some of my old test code for running Ogg media on iOS, adding WebM support and turning it into OGVKit, a (soon-to-be) reusable library that we can use to finally add video and audio playback to our Wikipedia iPhone app.

Of course decoding VP8 or Theora video on the CPU is going to be more expensive in terms of energy usage than decoding H.264 in dedicated silicon… but how much more?

The iOS 9 beta SDK supports enhanced energy monitoring in Xcode 7 beta… let’s try it out! The diagnostic detail screen looks like so:

energy - whole

Whoa! That’s a little overwhelming. What’s actually going on here?

First, what’s going on here

I’ve got my OGVKit demo app playing this video “Curiosity’s Seven Minutes of Terror” found on Wikimedia Commons, on two devices running iOS 9 beta: an iPod Touch (the lowest-end currently sold iDevice) and an iPad Air (one generation behind the highest-end currently sold iDevice).

The iPod Touch is playing a modest 360p WebM transcode, while the iPad Air is playing a higher-resolution 720p WebM transcode with its beefier 64-bit CPU:

energy - wassupFirst look: the cost of networking

At first, the energy usage looks pretty high:

energy - network highThis however is because in addition to media playback we’re buffering umpty-ump megabytes over HTTPS over wifi — as fast as a 150 Mbps cable connection will allow.

energy - before n afterOnce the download completes, the CPU usage from SSL decoding goes down, the wifi reduces its power consumption, and our energy usage relatively flattens.

Now what’s the spot-meter look like?

energy - lowPretty cool, right!?

See approximate reported energy usage levels for all transcode formats (Ogg Theora and WebM at various resolutions) if you like! Ogg Theora is a little faster to decode but WebM looks significantly better at the bitrates we use.

Ok but how’s that compare to native H.264 playback?

Good question. I’m about to try it and find out.

….

Ok here’s what we got:

energy - mp4The native AVPlayer downloads smaller chunks more slowly, but similarly shows higher CPU and energy usage during download. Once playing only, reported CPU usage dives to a percent or two and the reported energy impact is “Zero”.

Now, I’m not sure I believe “Zero”… ūüėČ

I suppose I’ll have to rig up some kind of ‘run until the battery dies’ test to compare how reasonable this looks for non-trivial playback times… but the ‘Low’ reportage for WebM at reasonable resolutions makes me happier than ‘Very High’ would have!

MediaWiki audio/video support updates

Mirror of mailing list post on wikitech-l and related lists

I’ve been passing the last few days feverishly working on audio/video stuff, cause it’s been driving me nuts that it’s not quite in working shape.

TL;DR: Major fixes in the works for Android, Safari (iOS and Mac), and IE/Edge (Windows). Need testers and patch reviewers.

ogv.js for Safari/IE/Edge

In recent versions of Safari, Internet Explorer, and Microsoft’s upcoming Edge browser, there’s still no default Ogg or WebM support but JavaScript has gotten fast enough to run an Ogg Theora/Vorbis decoder with CPU to spare for drawing and outputting sound in real time.

The ogv.js decoder/player has been one of my fun projects for some time, and I think I’m finally happy with my TimedMediaHandler/MwEmbedPlayer integration patch for the desktop MediaWiki interface.

I’ll want to update it to work with Video.js later, but I’d love to get this version reviewed and deployed in the meantime.

Please head over to https://ogvjs-testing.wmflabs.org/ in Safari 6.1+ or IE 10+ (or ‘Project Spartan’ on Windows 10 preview) and try it out! Particularly interested in cases where it doesn’t work or messes up.

Non-JavaScript fallback for iOS

I’ve found that Safari on iOS supports QuickTime movies with Motion-JPEG video and mu-law PCM audio. JPEG and PCM are, as it happens, old and not so much patented. \o/

As such this should work as a fallback for basic audio and video on older iPhones and iPads that can’t run ogv.js well, or in web views in apps that use Apple’s older web embedding APIs where JavaScript is slow (for example, Chrome for iOS).

However these get really bad compression ratios, so to keep bandwidth down similar to the 360p Ogg and WebM versions I had to reduce quality and resolution significantly. Hold an iPhone at arm’s length and it’s maybe ok, but zoom full-screen on your iPad and you’ll hate the giant blurry pixels!

This should also provide a working basic audio/video experience in our Wikipedia iOS app, until such time as we integrate Ogg or WebM decoding natively into the app.

Note that it seems tricky to bulk-run new transcodes on old files with TimedMediaHandler. I assume there’s a convenient way to do it that I just haven’t found in the extension maint scripts…

In progress: mobile video fixes

Audio has worked on Android for a while — the .ogg files show up in native <audio> elements and Just Work.

But video has been often broken, with TimedMediaHandler’s “popup transforms” reducing most video embeds into a thumbnail and a link to the original file — which might play if WebM (not if Ogg Theora) but it might also be a 1080p original which you don’t want to pull down on 3G! And neither audio nor video has worked on iOS.

This patch adds a simple mobile target for TMH, which fixes the popup transforms to look better and actually work by loading up an embedded-size player with the appropriately playable transcodes (WebM, Ogg, and the MJPEG last-ditch fallback).

ogv.js is used if available and necessary, for instance in iOS Safari when the CPU is fast enough. (Known to work only on 64-bit models.)

Future: codec.js and WebM and OGVKit

For the future, I’m also working on extending ogv.js to support WebM for better quality (especially in high-motion scenes) — once that stabilizes I’ll rename the combined package codec.js. Performance of WebM is not yet good enough to deploy, and some features like seeking are still missing, but breaking out the codec modules means I can develop the codecs in parallel and keep the high-level player logic in common.

Browser infrastructure improvements like SIMD, threading, and more GPU access should continue to make WebM decoding faster in the future as well.

I’d also like to finish up my OGVKit package for iOS, so we can embed a basic audio/video player at full quality into the Wikipedia iOS app. This needs some more cleanup work still.

Phew! Ok that’s about it.

im in ur javascript, decoding ur webm

Been a while since I posted; working on various bits and bobs.

Most exciting at the moment: initial work on pure-JavaScript WebM playback support for ogv.js (soon to become codec.js!)

This is currently very incomplete; video doesn’t come out right and audio’s not hooked up yet, and most notably you only get a few frames before it craps out. ūüôā

Update: Got audio working after some sleep, and playback is more thorough. Yay! Still no seeking yet, and playback may not always reach completion.

I’ve seen a couple other attempts to build WebM decoders in JS using emscripten (eg Route9), but nobody’s gotten them hooked up to a full-blown player with audio yet. With a little more work on the decoder side, I should be able to leverage all of my investment in ogv.js’s player logic.

webm-in-progresslive preview of current experimental build

How’s it work?

I’m using nestegg for the WebM container parsing; currently this is giving me some impedence mismatch with the first-generation ogv.js code due to different i/o models.

ogv.js’s C-side Ogg container parsing was set up with entirely asynchronous i/o due to the needs of using async XHR to fetch data over the network. Nestegg, however, uses synchronous i/o callbacks in some places, and I really don’t want to jump through all the hoops to do WebM cue seeking without using the library. (Already did that with Ogg and it was horrid!)

I may be able to adapt the ogv.js C-side code to using synchronous i/o callbacks and let them get mapped to async i/o on the JS side, thanks to some fancy features in emscripten — either Asyncify or the newer emterpreter mode for parts of the high-level code while keeping the low-level decoder in pure asm.js. Need to do a little more research… ūüôā

Possibilities

Performance: speed on WebM/VP8 decoding won’t be as good as the Ogg/Theora decoder for now so may not be suitable for mobile devices, but should work fine on many laptop/desktop machines.

VP9: untested so far, but should work… though again, likely to be still slower than VP8.

Threading: currently threading is disabled in the build. There’s work ongoing at Mozilla to support pthreads for emscripten programs, using a new SharedArrayBuffer interface to share memory between Web Worker threads. Might experiment with that later as libvpx has some threading options (may only be useful for VP9 multi-slice decoding for now) but it’ll be a while before most browsers support it. However this might be a good use case for asking Microsoft and Apple to add support. ūüôā

SIMD: there’s also ongoing work on SIMD instruction abstraction in JavaScript, with Mozilla and Microsoft partnering. Will have to test with autovectorization in the compiler enabled and see if it makes any difference to decode speed, but don’t know how complete the support in Firefox and Edge is etc.

ogv.js MediaWiki integration updates

Over the last few weekends I’ve continued to poke at ogv.js, both the core library and the experimental MediaWiki integration. It’s getting pretty close to merge-ready!

Recent improvements to ogv.js player (gerrit changeset):

  • Audio no longer super-choppy in background tabs
  • ‘ended’ is no longer unreasonably delayed
  • various code cleanup
  • ogvjs-version.js with build timestamp available for use as a cache-buster helper

Fixes to the MediaWiki TimedMediaHandler desktop player integration (gerrit changeset):

  • Post-playback behavior is now the same as when using native playback
  • Various code cleanup

Fixes to the MediaWiki MobileFrontend mobile player integration (gerrit changeset):

  • Autoplay now working with native playback in Chrome and Firefox
  • Updated to work with current MobileFrontend (internal API changes)
  • Mobile media overlay now directly inherits from the MobileFrontend photo overlay class instead of duplicating it
  • Slow-CPU check is now applied on mobile player — this gets ogv.js video at 160p working on an old iPhone 4S running iOS 7! Fast A7-based iPhones/iPads still get 360p.

While we’re at it, Microsoft is opening up a public ‘suggestion box’ for Internet Explorer — folks might want to put in their votes for native Ogg Vorbis/Theora and WebM playback.

Testing ogv.js in MediaWiki

After many weekends of development on the¬†ogv.js¬†JavaScript Ogg Vorbis/Theora player¬†I’ve started work on embedding it¬†as a player into MediaWiki’s TimedMediaHandler extension.

The JavaScript version is functional in Safari and IE 10/11, though there’s some work yet to be done…¬†See a live wiki at ogvjs-testing.wmflabs.org or the in-progress patch set.

Screen Shot 2014-07-13 at 8.43.34 PM

 

Meanwhile, stay tuned during the week for some demos of the soon-to-be-majorly-updated Wikipedia iOS app!

OgvKit: native Ogg Vorbis/Theora playing on iOS

In addition to my in-browser ogv.js media player, I’ve got an OgvKit framework in progress for native iPhone and iPad apps, which I hope to integrate into Wikipedia’s new iOS app somewhere down the line.¬†I took a little¬†Independence Day holiday time and made a bunch of improvements¬†from where I last left it a few months ago:

(If the video is sideways, sorry — it’s still “processing” as of this writing.)

  • Color conversion¬†is OpenGL ES-accelerated, cutting CPU usage¬†in half when playing video.
  • Audio output actually works, more or less in¬†sync.
  • Framework¬†now packaged as a Cocoa Touch Static Library project

Shiny new A7-based 64-bit devices play 480p and even 720p¬†happily (iPhone 5S, iPad Air) but still¬†struggle with some 1080p originals. My oldest devices like an iPhone 3Gs and iPod Touch 4th-gen can’t go higher than 160p or so at present — still significantly faster than the JavaScript ogv.js version which can’t handle video on those devices at all.

Future work for another weekend:

  • It turns out¬†the xiph.org git mirrors of theora etc are¬†not being maintained; will switch fetching of library source to SVN.
  • Improved controls, seeking, fullscreen
  • Cache downloaded data on disk instead of RAM!
  • Move more of the player code from the demo into the library so it can be reused…
  • Test standalone packaging of the library so it can be dropped in to other projects easily
  • Try to get assembly¬†in libtheora to¬†build, and/or replace the ARM assembly code with C SIMD intrinsics
  • Find and test on an armv7s device (iPhone 5, 5C or iPad 4th-generation)