Canvas, Web Audio, MediaStream oh my!

I’ve often wished that for ogv.js I could send my raw video and audio output directly to a “real” <video> element for rendering instead of drawing on a <canvas> and playing sound separately to a Web Audio context.

In particular, things I want:

  • Not having to convert YUV to RGB myself
  • Not having to replicate the behavior of a <video> element’s sizing!
  • The warm fuzzy feeling of semantic correctness
  • Making use of browser extensions like control buttons for an active video element
  • Being able to use browser extensions like sending output to ChromeCast or AirPlay
  • Disabling screen dimming/lock during playback

This last is especially important for videos of non-trivial length, especially on phones which often have very aggressive screen dimming timeouts.

Well, in some browsers (Chrome and Firefox) now you can do at least some of this. 🙂

I’ve done a quick experiment using the <canvas> element’s captureStream() method to capture the video output — plus a capture node on the Web Audio graph — combining the two separate streams into a single MediaStream, and then piping that into a <video> for playback. Still have to do YUV to RGB conversion myself, but final output goes into an honest-to-gosh <video> element.

To my great pleasure it works! Though in Firefox I have some flickering that may be a bug, I’ll have to track it down.

Some issues:

  • Flickering on Firefox. Might just be my GPU, might be something else.
  • The <video> doesn’t have insight to things like duration, seeking, etc, so can’t rely on native controls or API of the <video> alone acting like a native <video> with a file source.
  • Pretty sure there are inefficiencies. Have not tested performance or checked if there’s double YUV->RGB->YUV->RGB going on.

Of course, Chrome and Firefox are the browsers I don’t need ogv.js for for Wikipedia’s current usage, since they play WebM and Ogg natively already. But if Safari and Edge adopt the necessary interfaces and WebRTC-related infrastructure for MediaStreams, it might become possible to use Safari’s full screen view, AirPlay mirroring, and picture-in-picture with ogv.js-driven playback of Ogg, WebM, and potentially other custom or legacy or niche formats.

Unfortunately I can’t test whether casting to a ChromeCast works in Chrome as I’m traveling and don’t have one handy just now. Hoping to find out soon! 😀

HDTV and the video look

I spent some time last night playing with my parents’ shiny new HDTV, which puts my 2005-vintage 26″ set to shame.

Pretty nice set; 40-something inch, 1080p, 120 Hz whatchamahooie, and you can plug in a USB stick full of JPEGs and force your family to watch your vacation photos. Nice!

It seems to be all the rage on new sets to have motion interpolation which can take 24-frame-sourced content (feature films and most US drama and sitcom TV shows) and smooth out the frame-to-frame motion, making it look more like 60-field video. Lots of higher-end sets advertise 120 Hz or even 240 Hz, which honestly seems excessive to me — the human eye can’t distinguish much more than 60 frames per second. 🙂

I’m a bit torn; on the one hand, the faster frame rate makes motion look much more vivid and realistic from any objective point of view. On the other hand, audiences have been trained over the last few decades to associate the video look with “cheesier” programming — soaps, reality shows, etc — while “serious” programs are shot on film at 24fps, making them feel more like a big-budget feature film… even to the point that lots of money was spent developing HD video cameras that could shoot at the slower, less realistic 24fps instead of HD’s native 60!

We stumbled into Harold and Kumar escape from Guantanamo of all things on HBO, and ran it for a while just to get a feel for the set. At first it drove me nuts seeing a movie I’d already seen on film looking distinctly like HD video, but after a half hour I got quite used to it and rather grew to like it. Of course as a former cinema-television student I’m extra-sensitized to this stuff — my wife immediately took to the more vivid display and commented on how much better it looked than when we’d seen it in the theater!

Looks like the mass audiences are happy to embrace high-motion video… I wonder if the long-standing holdover of the “film look” over the last decade was driven more by the oversensitized film geeks in the industry than any actual audience comparison…

Let’s learn a lesson here with our software development as well — those of us who’ve been nose-deep in web sites and software UI for years aren’t necessarily the most qualified to tell what our actual users are going to be most comfortable with.