Canvas, Web Audio, MediaStream oh my!

I’ve often wished that for ogv.js I could send my raw video and audio output directly to a “real” <video> element for rendering instead of drawing on a <canvas> and playing sound separately to a Web Audio context.

In particular, things I want:

  • Not having to convert YUV to RGB myself
  • Not having to replicate the behavior of a <video> element’s sizing!
  • The warm fuzzy feeling of semantic correctness
  • Making use of browser extensions like control buttons for an active video element
  • Being able to use browser extensions like sending output to ChromeCast or AirPlay
  • Disabling screen dimming/lock during playback

This last is especially important for videos of non-trivial length, especially on phones which often have very aggressive screen dimming timeouts.

Well, in some browsers (Chrome and Firefox) now you can do at least some of this. :)

I’ve done a quick experiment using the <canvas> element’s captureStream() method to capture the video output — plus a capture node on the Web Audio graph — combining the two separate streams into a single MediaStream, and then piping that into a <video> for playback. Still have to do YUV to RGB conversion myself, but final output goes into an honest-to-gosh <video> element.

To my great pleasure it works! Though in Firefox I have some flickering that may be a bug, I’ll have to track it down.

Some issues:

  • Flickering on Firefox. Might just be my GPU, might be something else.
  • The <video> doesn’t have insight to things like duration, seeking, etc, so can’t rely on native controls or API of the <video> alone acting like a native <video> with a file source.
  • Pretty sure there are inefficiencies. Have not tested performance or checked if there’s double YUV->RGB->YUV->RGB going on.

Of course, Chrome and Firefox are the browsers I don’t need ogv.js for for Wikipedia’s current usage, since they play WebM and Ogg natively already. But if Safari and Edge adopt the necessary interfaces and WebRTC-related infrastructure for MediaStreams, it might become possible to use Safari’s full screen view, AirPlay mirroring, and picture-in-picture with ogv.js-driven playback of Ogg, WebM, and potentially other custom or legacy or niche formats.

Unfortunately I can’t test whether casting to a ChromeCast works in Chrome as I’m traveling and don’t have one handy just now. Hoping to find out soon! :D

Alliance for Open Media code drop & more hardware partners

Very exciting! The new video codec so far is mostly based on Google’s in-development VP10 (next gen of the VP8/VP9 used in WebM) but is being co-developed and supported with a number of other orgs.

  • CPU/GPU/SoC makers: Intel, AMD, ARM, NVidia
  • OS & machine makers: Google, Microsoft, Cisco
  • Browser makers: Mozilla, Google, Microsoft
  • Content farms: Netflix, Google (YouTube)

Microsoft is also actively working on VP8/VP9 support for Windows 10, with some limited compatibility in preview releases.

As always, Apple remains conspicuously absent. :(

Like the earlier VP8/VP9 the patent licenses are open and don’t have the kind of weird clauses that have tripped up MPEGLA’s H.264 and HEVC/H.265 in some quarters. (*cough* Linux *cough* Wikipedia)

Totally trying to figure out how we can get involved at this stage; making sure I can build the codec in my iOS app and JavaScript shim environments will be a great start!

Popcorn Maker is dead, long live Popcorn Editor

One of the really cool ‘web maker’ projects that Mozilla sponsored in the last few years was Popcorn Maker, an in-browser video editor that could take direct video clips or videos from Youtube etc and let you remix to your heart’s content.

Obviously this capability is very attractive for the future of video on Wikipedia and other Wikimedia sites! (See Ben’s blog post from July and Mike’s presentation on Air Mozilla)

Unfortunately Mozilla has to shuffle priorities, and Popcorn Maker is on the outs.

Fortunately, there’s enough interest in the Wikimedia video world that we’re helping to pick it up!

Popcorn Editor is a standalone version of the Popcorn Maker editor tool, and I’m going to be helping with integrating that into MediaWiki. We’ve got a milestone in the bug tracker and everything. :)

If you’re interested in helping out, we’re going to have some work sprints at WikiConference USA next weekend in Washington, DC. Please come and help out!

 

Using Web Worker threading in ogv.js for smoother playback

I’ve been cleaning up MediaWiki’s “TimedMediaHandler” extension in preparation for merging integration with my ogv.js JavaScript media player for Safari and IE/Edge when no native WebM or Ogg playback is available. It’s coming along well, but one thing continued to worry me about performance: when things worked it was great, but if the video decode was too slow, it could make your browser very sluggish.

In addition to simply selecting too high a resolution for your CPU, this can strike if you accidentally open a debug console during playback (Safari, IE) or more seriously if you’re on an obscure platform that doesn’t have a JavaScript JIT compiler… or using an app’s embedded browser on your iPhone.

Or simply if the integration code’s automatic benchmark overestimated your browser speed, running too much decoding just made everything crap.

Luckily, the HTML5 web platform has a solution — Web Workers.

Workers provide a limited ability to do multithreading in JavaScript, which means the decoder thread can take as long as it wants — the main browser thread can keep responding to user input in the meantime.

The limitation is that scripts running in a Worker have no direct access to your code or data running in the web page’s main thread — you can communicate only by sending messages with ‘raw’ data types. Folks working with lots of DOM browser nodes thus can’t get much benefit, but for buffer-driven compute tasks like media decoding it’s perfect!

Threading comms overhead

My first attempt was to take the existing decoder class (an emscripten module containing the Ogg demuxer, Theora video decoder, and Vorbis audio decoder, etc) and run it in the worker thread, with a proxy object sending data and updated object properties back and forth.

This required a little refactoring to make the decoder interfaces asynchronous, taking callbacks instead of returning results immediately.

It worked pretty well, but there was a lot of overhead due to the demuxer requiring frequent back-and-forth calls — after every processing churn, we had to wait for the demuxer to return its updated status to us on the main thread.

This only took a fraction of a millisecond each time, but a bunch of those add up when your per-frame budget is 1/30 (or even 1/60) second!

 

I had been intending a bigger refactor of the code anyway to use separate emscripten modules for the demuxer and audio/video decoders — this means you don’t have to load code you won’t need, like the Opus audio decoder when you’re only playing Vorbis files.

It also means I could change the coupling, keeping the demuxer on the main thread and moving just the audio/video decoders to workers.

This gives me full speed going back-and-forth on the demuxer, while the decoders can switch to a more “streaming” behavior, sending packets down to be decoded and then displaying the frames or queueing the audio whenever it comes back, without having to wait on it for the next processing iteration.

The result is pretty awesome — in particular on older Windows machines, IE 11 has to use the Flash plugin to do audio and I was previously seeing a lot of “stuttery” behavior when the video decode blocked the Flash audio queueing or vice versa… now it’s much smoother.

The main bug left in the worker mode is that my audio/video sync handling code doesn’t properly handle the case where video decoding is consistently too slow — when we were on the main thread, this caused the audio to halt due to the main thread being blocked; now the audio just keeps on going and the video keeps playing as fast as it can and never catches up. :)

However this should be easy to fix, and having it be wrong but NOT FREEZING YOUR BROWSER is an improvement over having sync but FREEZING YOUR BROWSER. :)

ogv.js MediaWiki integration updates

Over the last few weekends I’ve continued to poke at ogv.js, both the core library and the experimental MediaWiki integration. It’s getting pretty close to merge-ready!

Recent improvements to ogv.js player (gerrit changeset):

  • Audio no longer super-choppy in background tabs
  • ‘ended’ is no longer unreasonably delayed
  • various code cleanup
  • ogvjs-version.js with build timestamp available for use as a cache-buster helper

Fixes to the MediaWiki TimedMediaHandler desktop player integration (gerrit changeset):

  • Post-playback behavior is now the same as when using native playback
  • Various code cleanup

Fixes to the MediaWiki MobileFrontend mobile player integration (gerrit changeset):

  • Autoplay now working with native playback in Chrome and Firefox
  • Updated to work with current MobileFrontend (internal API changes)
  • Mobile media overlay now directly inherits from the MobileFrontend photo overlay class instead of duplicating it
  • Slow-CPU check is now applied on mobile player — this gets ogv.js video at 160p working on an old iPhone 4S running iOS 7! Fast A7-based iPhones/iPads still get 360p.

While we’re at it, Microsoft is opening up a public ‘suggestion box’ for Internet Explorer — folks might want to put in their votes for native Ogg Vorbis/Theora and WebM playback.

Flame on! Trying out Firefox OS at 2.1…

Ever since I heard about Mozilla’s ‘Boot2Gecko‘ project a few years back I was very excited about the eventual possibility of Firefox-powered phones running a truly free operating system, with apps provided through the open web  instead of platform-lock-in walled gardens.

It’s been a long journey though, and often a painful one. Early versions of Firefox OS were pretty rough, it was hard to get phones that weren’t severely underpowered, and actually upgrading to the latest versions on a release phone was….. often not really possible.

So I finally gave in and picked up the Flame, which is the officially recommended Firefox OS reference device. Current builds are actually, like, published for it!

I immediately flashed the device to the current base image (v180, with a low-level ‘Gonk’ layer based on Android 4.4’s low-level Linux layers) and updated to the almost-ready-for-release Firefox OS 2.1.

Version 2.1 finally does away with the old crappy browser app and treats web site browsing on the same level as installed ‘apps’. Graphics are pretty smooth, using hardware compositing, and in general it’s a HUGE improvement over 1.x.

The Flame

Hardware notes

    • The Flame is meant to be representative of the next generation of Firefox OS release phones which are targeting developing markets, so it’s not as fancy as the latest Android or iOS devices.
      • The screen is only 1.5x density, versus 3x on my Nexus 5. But it’s still a big improvement over the older 1x 320×480 devices.
      • Decent 1GB RAM — can be configured lower to simulate lower-end devices, which I have not attempted. Eek!
      • There’s a limited amount of internal storage, and a micro-SD card slot where you’re expected to store additional files such as media. I only had a 4GB card handy from an old phone so I’m using that for now, but will replace it with a 32GB card later.
    • the Flame has 2 SIM slots, both full-size. This meant I needed a micro-SIM-to-fullsize-SIM adapter to get my main phone line running on the Flame. The micro-SIM kept popping out of the adapter while trying to insert it, but I eventually got it in intact and it’s working fine. (T-Mobile US, HSDPA speeds. No LTE support on the Flame.) Conveniently the adapter kit also included the necessary adapter to move my backup/testing phone line from my iPhone 5s (nano-SIM) to the Nexus 5. Why can’t we all just use the same damn size SIM?

The camera seems kinda awful; video framerate is bad. Not sure if this is a software bug or a hardware limitation but it’s a bit of a bummer.

Back to the web: de-appifying

The most common apps I use on my Nexus 5 are:

    • Gmail
    • Google Maps
    • Facebook
    • Twitter
    • Feedly
    • Kindle
    • Amazon Music

Wikipedia

These are all available on the web, but with some caveats and limitations:

  • Gmail shows in a really crappy old-school mobile web interface instead of the nice modern HTML5 one you get on an Android or iOS device. I can’t seem to use it for multiple accounts either, which makes it a non-starter since I have both personal (gmail) and work (gapps) accounts. I’ve been using the Firefox OS built-in Email app instead for now, which seems to work better than in old versions but isn’t really optimized for my ‘archive everything’ workflow.
  • Google Maps shows the web interface, which is kinda ugly but seems to work including geolocation and transit directions. YAYYY
  • Facebook web seems pretty decent at least for reading, but I don’t get notifications of replies and have to check manually.
  • Twitter web seems pretty good, though the pull-to-refresh is a little flaky and again no notifications.
  • Feedly’s web interface is designed for desktop screens and doesn’t scale down property to a smartphone screen. BOOO
  • Kindle Cloud Reader actually runs — it downloads and views books and everything. But again, it’s designed for desktop and tablet screens and the UI doesn’t scale down. You can only see the top-left corner of the page and can’t actually read anything. BOOOOO
  • Amazon Cloud Player for online-stored music….. amazingly this works, but the interface is desktop-oriented and distinctly NOT mobile friendly. (It also prompts for Adobe Flash, but doesn’t seem to require it for playback.) However since playback stops when you switch away from the app, it’s kind of a bummer to use. BOOOO
  • We have a Firefox OS Wikipedia reader app based on our old PhoneGap app — it works fine, but hasn’t been maintained much and needs an overhaul. Meanwhile our mobile web site also works pretty well on Firefox os, and now supports editing and all kinds of cute stuff. YAYYYY

Now, some things I can’t get at all:

  • Uber
  • Skype
  • Walgreens

:(

  • There’s really nothing in Uber that needs to be an app as a customer — they could just as easily have a web app with all the capabilities of looking up, calling a car, watching the map, etc. I can’t even successfully log in to their web interface for viewing my past rides, but if I could there’s no way to call a cab there.
  • I occasionally use Skype, mainly when XBox Live’s chat system breaks. *cough* Microsoft owns them both *cough*. That’s all native apps and has no web option.
  • The Walgreens app on iOS/Android lets you scan the barcode on your medication to schedule a refill, it’s pretty handy! Their web site has no equivalent that I can find… but I can work around it by renewing via email notification instead.

So I’ll be carrying the Flame around as my main phone line for at least a bit, but I’m gonna keep the Nexus 5 around for a few things.

We’ll see how long it takes before I switch the main line back to Android or if I stick with it. :D

People should be allowed to be wrong

I support marriage equality, and I am deeply troubled by the tone of recent discourse in the Mozilla community.

People should be allowed to be wrong. People should be capable of working together on common goals despite their differences in other areas.

I appreciate that in California us “liberals” and “progressives” have things pretty good and can afford to take on long-neglected issues like equality of legal marriage rights for everybody in a loving, committed relationship regardless of gender or sexuality. I think that’s great, and I’m proud to live in a state that has been willing to take on the issue. It feels like we live at the forefront of history on basic freedoms for all sorts of often hated and misunderstood groups.

But the world isn’t a single-issue voter. The world is BILLIONS OF PEOPLE, most of whom believe things you or I find highly offensive and vice-versa.

If the tables were turned it might easily be atheists like me, or gays like many of my colleagues and friends who are on the wrong end of the stick — as we and they often have been for generations.

I choose instead to reject the stick as a tool and embrace tolerance and forgiveness.

Some may accuse me of false equivalence. All I know is that destroying someone’s career doesn’t feel like a victory.

Now can we all get back to writing software that improves peoples’ lives?

ogv.js update: color, sound

Last week I posted about a proof of concept experiment cross-compiling the Theora video decoder to JavaScript: ogv.js.

Well, I did some more hacking on it this weekend:

  • Color output? Check.
  • Streaming input? Check. (But no seeking yet, and buffering is aggressive on some browsers.)
  • Sound? Check. (But it’s not synced, choppy, and usually the wrong sample rate. And no IE or iOS support yet.)
  • Pretty interface to select any file from Wikimedia Commons’ Media of the Day archive? Check.
  • Laid some groundwork for separating the codec to a background Worker thread (not yet implemented)
  • See the readme for more info.

Try out the live demo with this pretty video of a Montreal subway station:

Jarry metro video screenshot

Feel free to try building or hacking the source for fun. :)

Firefox OS test phone!

So we got a Firefox OS test phone at Wikimedia, and of course as the resident Firefox-lover I took it for testing. :)

This isn’t the final hardware — it’s a reflashed Android phone — but apparently specs are close.

Performance

Mozilla is currently aiming at the lower-end market in developing nations rather than trying to battle it out with Apple and Samsung etc at the high-end. The test device is a lower-end Android phone (built by ZTE for the Turkish market) with a medium-resolution screen (320×480) like the pre-Retina iPhone, and a slower processor than the latest whiz-bang phones have.

UI smoothness ranges from “pretty good” to “sluggish” depending on what’s running. There’s definitely still room for optimization and I’m hoping to see performance improve, but it’s usable enough for a low-end device.

I can’t seem to get on T-Mobile’s 3G network, so I’m stuck on Edge when out of wifi range… brings back memories of my original-model iPhone. :)

UI style

Firefox OS uses a single hardware “home” button, like iOS. The test device has the full set of 4 Android hardware buttons, so I had to get out of the habit of going for the ‘back’ button… but this won’t be a problem on final devices. Long-pressing the home button opens a “card view” application switcher, which reminds me of WebOS or the BlackBerry PlayBook — you can explicitly close out an app by flicking it towards the top of the screen.

The home screen is relatively straightforward; there’s an iOS-like page grid of applications off to the right; on the left is an “everything.me” tab with lots of links to popular web sites… none of which load. I’m sure this will be fixed shortly. In the meantime I’m ignoring those links and using the browser, installable apps, or making my own homescreen bookmarks from the browser.

The browser app has a nice proper Firefox icon and looks much like Firefox on Android but with fewer options. Multiple tabs are available as expected from the upper-right corner, and back/forward and bookmark buttons are available on a toolbar at the bottom. There’s a bug where sometimes the toolbar covers up part of a web page, but it looks like this is being fixed.

You can bookmark a page either within the browser’s bookmarks or to the homescreen — a homescreen bookmark opens fullscreen like a standalone app (but with a collapsed toolbar available for back/forward navigation) instead of in the browser app. This is usually ok, but links opened from those apps tend to open in a “popup”-like closable window instead of opening the browser… so you can’t bookmark an opened link easily. :(

App lifecycle

An ‘application’ on Firefox OS is basically a web page running in an isolated process… you can kind of think of it like separate browser tabs, sorta. It’s a clean-sounding model, but there are some lifecycle issues.

When memory runs low, background apps may get closed. As far as I can tell they don’t get any sort of warning or have a chance to set relative priority; I’ve seen the Music app get auto-closed while playing music because I opened two other apps, which is kinda not cool.

In contrast, iOS for instance sends explicit low-memory warnings to apps, and gives a backgrounding app the chance to declare that it’s running audio or a download that needs to continue… Android also seems to not kill active background tasks so aggressively in my experience.

Hopefully this will be improved… added a note on a bug.

Apps don’t seem to be very consistent about saving state when they background either, so for instance you can’t just reopen Music and have it continue from where you left off. :(

Firefox Sync

Sync appears to be missing currently. This is a shame, as bookmark/history/tab/password syncing between desktop/laptop devices and with my Android devices is the killer feature that gets me to use Firefox on a phone.

Hopefully this will come in the future… filed an enhancement bug.

Offline access

Most built-in apps appear to be stored on the actual phone, but some like the Marketplace must be online and will just fail out when loaded offline.

A lot of the third-party apps I’ve tried, including games and utilities, seem to be hosted on web sites and don’t have offline manifests, so you can’t play Galactians on the subway unless it’s been cached recently. :(

App developers will need to get used to doing offline AppCache manifests properly…

Software to live by

I picked up a prepaid T-Mobile SIM card so I can use the test phone on my daily commute. This is about 10-15 minutes of walking, 10-15 minutes of streetcar, and then another few minutes in the subway — a nice workout for wireless connectivity and disconnectivity…

  • We’re still working on our Wikipedia app, ported over from the PhoneGap-based Android/iOS one — I have a copy running at http://ff.leuksman.com/ but it doesn’t work offline yet. Needs a little more work. :) Our mobile web interface also works just fine, and (since we made some recent fixes) correctly forwards you if you go to the desktop site.
  • NextBus works — though the GPS is a bit slow on this device — but doesn’t automatically detect that it’s a smartphone. Filed an evangelism bug.
  • Gmail loads up in desktop mode, which is pretty unusable on a tiny screen. Filed an evangelism bug, but not much hope right now as Firefox for Android gets a horrible primitive interface instead of the nice one that iOS and Android browser/Chrome get. :(
  • Also tried setting up my Gmail account with the built-in mail app; initially had some problems with the setup but got it working. Since I’m using 2-factor auth I had to set up an app-specific password. Sends and receives ok, but the interface is a bit primitive and I can’t easily switch to use my alternate ‘from’ addresses.
  • Twitter has an official app available in the Mozilla Marketplace. It doesn’t seem to work offline, but for catching up on friends’ tweets while connectivity is there it seems to work just fine and looks nice.
  • Facebook doesn’t have an app, but its touch-friendly web interface now works pretty nicely in Firefox, and can be easily bookmarked to the home screen. There are a few bugs on the board (link me) but I’ve had no troubles reading and posting basic updates. Facebook’s apps don’t work offline anyway, so this seems fine.
  • There’s not a default note-taking application… I ended up installing a little open-source app ‘Prio Note’ which lets you save short notes and filter by priority; handy enough for typing notes like “Kindle didn’t work” but it’s a bit flaky offline. On the plus side it’s open source so I may submit patches. ;)
  • There’s also an official Evernote “app” in the Marketplace… but it just points to their web interface, which doesn’t fit on a mobile phone screen once you get past the sign-in screen. Gave them a 1-star review. >:(
  • I tend to use my phones for reading, so I went looking for a Kindle app… no app available, but Kindle Cloud Reader sortof works. Unfortunately it’s designed for tablet and laptop/desktop screens and the UI doesn’t fit on the phone, so it’s not actually usable although I can download books and read them (as long as I only need the upper-left quarter of the page). With some UI work by Amazon this could make a fine Firefox OS app though.
  • Pandora has no app, and the web interface should work but just kicks out an “upgrade your browser or Flash” message. :( As an alternative, there’s a built-in app that’s an FM radio tuner. o_O Remember radio? It’s like Pandora but you can’t skip bad songs. ;) Filed an evangelism bug.
  • Built-in Music app plays .mp3 and (non-DRM’ed) .m4a files copied to the SD card with no trouble. Whee! UI is a little sluggish but works… but often pauses or dies while in the background.

 

Debugging

There’s some way to hook up the Firefox web inspector and debugger but I haven’t gotten into that level of detail yet. You can use the Android SDK tools for some basic things like watching the device log, taking screen shots, and getting a shell.

Storage

This device has a couple free gigs of device storage (for apps and data) and a micro-SD card with a few gigs for media storage — photos and videos taken with the camera save to the SD card, and the Music app looks for music there.

The SD card can be mounted over USB by enabling USB Mass Storage, and all seems to work as you expect — iPhoto on your Mac will slurp in the images, or you can copy files manually, whatever.

As a phone

Seems to work — I’ve made some calls to myself and it makes noise. :) Plan to use it some more just to try it out, but I’m not a big phone talker.