AudioFeeder updates for ogv.js

I’ve taken a break from the blog for too long! Time to update some on current work. We’re doing a final push on the video.js-based frontend media player for MediaWiki’s TimedMediaHandler, with some new user interface bits, better mobile support, and laying the foundation for smoother streaming in the future.

Among other things, I’m doing some cleanup on the AudioFeeder component in the ogv.js codec shim, which is still used in Safari on iOS devices and older Macs.

This abstracts a digital sound output channel with an append-only buffer, which can be stopped/started, the volume changed, and the current playback position queried.

When I was starting on this work in 2014 or so, Internet Explorer 11 was supported so I needed a Flash backend for IE, and a Web Audio backend for Safari… at the time, the only way to create the endless virtual buffer in Web Audio was using a ScriptProcessorNode, which ran its data-manipulation callback on the main thread. This required a fairly large buffer size for each callback to ensure the browser’s audio thread had data available if the main thread was hung up on drawing a video frame or something.

Fast forward to 2022: IE 11 and Flash are EOL and I’ve been able to drop them from our support matrix. Safari and other browsers still support ScriptProcessorNode, but it’s been officially deprecated for a while in favor of AudioWorklets.

I’ve been meaning to look into upgrading with an AudioWorklet backend but hadn’t had need; however I’m seeing some performance problems with the current code on Safari Technology Preview, especially on my slower 2015 MacBook Pro which doesn’t grok VP9 in hardware so needs the shim. :) Figured it’s worth taking a day or two to see if I can avoid a perf regression on old Macs when the next Safari update comes out.

So first — what’s a worklet? This is an interface that’s being adopted by a few web bits (I think some CSS animation bits are using these too) to have a fairly structured way of loading little scripts into a dedicated worker thread (the worklet) to do specific things off-main-thread that are performance critical (audio, layout, animation).

An AudioWorkletNode hooks into the Web Audio graph, giving something similar to a ScriptProcessorNode but where the processing callback runs in the worklet, on an AudioWorkletProcessor subclass. The processor object has audio-specific stuff like the media time at the start of the buffer, and is given input/output channels to read/write.

For an ever-growing output, we use 0 inputs and 1 output; ideally I can support multichannel audio as well, which I never bothered to do in the old code (for simplicity it downmixed everything to stereo). Because the worklet processors run on a dedicated thread, the data comes in small chunks — by default something like 128 samples — whereas I’d been using like 8192-sample buffers on the main thread! This allows you to have low latency, if you prefer it over a comfy buffer.

Communicating with worker threads traditionally sucks in JavaScript — unless you opt into the new secure stuff for shared memory buffers you have to send asynchronous messages; however those messages can include structured data so it’s easy to send Float32Arrays full of samples around.

The AudioWorkletNode on the main thread gets its own MessagePort, which connects to a fellow MessagePort on the AudioWorkletProcessor in the audio thread, and you can post JS objects back and forth, using the standard “structured clone” algorithm for stripping out local state.

I haven’t quite got it running yet but I think I’m close. ;) On node creation, an initial set of queued buffers are sent in with the setup parameters. When audio starts playing, after the first callback copies out its data it posts some state info back to the main thread, with the audio chunk’s timestamp and the number of samples output so far.

The main thread’s AudioFeeder abstraction can then pair those up to report what timestamp within the data feed is being played now, with compensation for any surprises (like the audio thread missing a callback itself, or a buffer underrun from the main thread causing a delay).

When stopping, instead of just removing the node from the audio graph, I’ve got the main thread sending down a message that notifies the worklet code that it can safely stop, and asking for any remaining data back. This is important if we’ve maintained a healthy buffer of decoded audio; in case we continue playback from the same position, we can pass the buffers back into the new worklet node.

I kinda like the interface now that I’m digging in it. Should work… Either tonight or tomorrow hope to sort that out and get ogv.js updated in-tree again in TMH.

SVN update on Wikimedia sites, questions on branching

I’ve got MediaWiki on the live Wikimedia sites all up to date at r43514 now.

We’d been updating some extensions individually over the last couple weeks, but full updates were held back as general code review got a little behind during the lead-up to the fundraiser and our staff meeting last week… I’m hoping to get us on a regular weekly update schedule, probably Tuesdays since my experience is that Mondays end up totally unproductive. :)

I did have to pull back the Special:Search redesign for the moment; it’s looking *awesome* but has a few glitches still, which I’m hoping we can resolve before putting it live.

See my comments on issues I noticed in the revert.

I’m thinking we should start making more active use of branches for experimental/iterative development like this, where existing features in core are majorly refactored and need some iterations of testing before going live.

We try to keep our trunk code ready-to-run at all times, so when something in trunk is not quite ready yet we end up rolling it back (which requires tracking down multiple changes and reverting all of them) or else rushing fixes so we can get an update pushed out.

The SVN server was updated to 1.5 a while ago, which is apparently a little handier at branch merging, but branching still is kind of awkward in SVN. Any good recommendations on SVN-friendly DVCSs? I know some folks use SVK or a GIT-SVN bridge for doing various local development, but how easy is it to share a development branch among multiple developers over time this way?

CodeReview live on

One of the problems we’ve been seeing is that our code review procedure doesn’t always scale well. We have a fairly large number of committers, and a pretty liberal policy about committing new code to trunk — but we also need things to work consistently so we can keep the production code up to date.

Traditionally, code that’s been committed to SVN gets reviewed offline by me or Tim before we push things out live. If we find problems, we fix them up or revert the code to be redone correctly later.

There’s a couple big problems with this:

  1. We can’t easily coordinate our notes; I can’t see what Tim’s reviewed and what he hasn’t. We end up either duplicating effort or missing things.
  2. If we’re both busy, sick, on vacation, etc sometimes it just doesn’t get done!
  3. If we want more people to pitch in, coordination gets even harder.

In my spare time over the last few weeks I’ve thrown together a little CodeReview extension for MediaWiki to help with this. It pulls the SVN revision data as commits are made and presents an interface on the wiki where we can see what’s been reviewed, tag problems, and add comments for follow-up issues.

Yesterday I went ahead and put it live:

The UI’s still a little rough, and not all linking and metadata features are implemented; Aaron’s going to help polish it up. :) But it’s already useful, and I’ve got some revisions flagged as fixme…

Feel free to try it out, and add notes for things which need work.

Currently comments are open to any registered user on the wiki; status changes and tagging updates are limited to the ‘coder’ group, which is viral — any coder can make another user a coder.