Scaling video playback on slow and fast CPUs in ogv.js

Video playback has different performance challenges at different scales, and mobile devices are a great place to see that in action. Nowhere is this more evident than in the iPhone/iPad lineup, where the same iOS 9.3 runs across several years worth of models with a huge variance in CPU speeds…

In ogv.js 1.1.2 I’ve got the threading using up to 3 threads at maximum utilization (iOS devices so far have only 2 cores): main thread, video decode thread, and audio decode thread. Handling of the decoded frames or audio packets is serialized through the main thread, where the player logic drives the demuxer, audio output, and frame blitting.

On the latest iPad Pro 9.7″, advertising “desktop-class performance”, I can play back the Blender sci-fi short Tears of Steel comfortably at 1080p24 in Ogg Theora:

The performance graph shows frames consistently on time (blue line is near the red target line) and a fair amount of headroom on the video decode thread (cyan) with a tiny amount of time spent on the audio thread (green) and main thread (black).

At this and higher resolutions, everything is dominated by video decode time — if we can keep up with it we’re golden, but if we get behind everything would ssllooww ddoownn badly.

On an iPad Air, two models behind, we get similar performance on the 720p24 version, at about half the pixels:

We can see the blue bars jumping up once a second, indicating sensitivity to the timing report and graph being updated once a second on the main thread, but overall still good. Audio in green is slightly higher but still ignorable.

On a much older iPad 3, another two models behind, we see a very different graph as we play back a mere 240p24 quarter-SD resolution file:

The iPad 3 has an older generation, 32-bit processor, and is in general pretty sluggish. Even at a low resolution, we have less headroom for the cyan bars of the video decode thread. Blue bars dipping below the red target line show we’re slipping on A/V sync sometimes. The green bars are much higher, indicating the audio decode thread is churning a lot harder to keep our buffers filled. Last but not least the gray bars at the bottom indicate more time spent in demuxing, drawing, etc on the main thread.

On this much slower processor, pushing audio decoding to another core makes a significant impact, saving an average of several milliseconds per frame by letting it overlap with video decoding.

The gray spikes from the main thread are from the demuxer, and after investigation turn out to be inflated by per-packet overhead on the tiny Vorbis audio packets… Such as adding timestamps to many of the packets. Ogg packs multiple small packets together into a single “page”, with only the final packet at the end of the page actually carrying a timestamp. Currently I’m using liboggz to encapsulate the demuxing, using its option to automatically calculate the missing timestamp deltas from header data in the packets… But this means every few frames the demuxer suddenly releases a burst of tiny packets with a 15-50ms delay on the main thread as it walks through them. On the slow end this can push a nearly late frame into late territory.

I may have further optimizations to make in keeping the main thread clear on slower CPUs, such as more efficient handling of download progress events, but overlapping the video and audio decode threads helps a lot.

On other machines like slow Windows boxes with blacklisted graphics drivers, we also benefit from firing off the next video decode before drawing the current frame — if WebGL is unexpectedly slow, or we fall back to CPU drawing, it may take a significant portion of our frame budget just to paint. Sending data down to the decode thread first means it’s more likely that the drawing won’t actually slow us down as much. This works wonders on a slow ARM-based Windows RT 8.1 Surface tablet. 🙂


A British driving adventure in five parts (or, Google Maps Can Suck It)

So I’m doing a little post-Wikimania traveling with my wife and my parents. Yesterday we drove from London to Cardiff via Stonehenge. It was… Quite the experience for a first-time driving in the UK.

Part one: Escaping London

IMG_0146.JPG Our adventure begins in the London Docklands, where the 72nd World Science Fiction Convention was held at the Excel Centre ( I was able to hire a car at the Europcar branch in the convention center, made it over to our hotel, and we just managed to squeeze our luggage into the back of this Skoda something or other.

Google Maps wanted to route us through the London city center to get out to the M4 motorway, but everyone I asked assured me this was a terrible idea and I should get to the M25 “orbital” highway that circles the city. A13 runs east from the docklands to the M25 and was pretty easy to get to; after some initial confusion getting used to driving on the left and being on the right side of the car I more or less adjusted, and we stopped for a quick lunch at a rest stop (“services centre”) off M25.

Part two: reaching Stonehenge

From there the route to Stonehenge was very simple: go south and west on the M25 orbital until the M3 branches off, then take A303 out to Amesbury and follow the signs to Stonehenge. This route was great; mostly big modern highways, well labeled, in the middle of the day. My main difficulty was adjusting to properly centering the car in the lane when I’m sitting on the “wrong” side of the car.


Part three: English country back road hell

When I planned out the route I didn’t do enough research on how to get back to the main motorway; it looked clear enough on Google Maps and I just turned on navigation on my phone and followed the directions a while.

The phone losing gps signal at first was a bad sign, but in retrospect the route was bad to begin with. We ended up taking A360 sorta northwestward toward the M4 which leads straight to Cardiff. As it turns out, while A303 was mostly a pretty comfortable minor highway, A360 is actually a series of tiny country and village back roads.

Often it narrows to one lane, has no shoulder, squirrels around and makes weird turns, etc. this was a somewhat harrowing experience, especially as signage was nearly nonexistent and I had a poor idea of how far I was from the main highway.

Part four: finding M4

Eventually we reached the entrance to M4… And I missed the exit from the roundabout and ended up on the wrong road. Google Maps rerouted us… Down another country road which eventually took us back to M4, much much later than I had hoped to be on the main road.

Once on M4 we were back in a world of wide lanes, divided highways, good signage, etc. Life was good again. We kept going west, crossing the Severn bridge to Wales. Interestingly this is a toll bridge westbound, but the toll collection is a good few miles past the bridge instead of before it like San Francisco’s bridges.

Part five: diversion hell

Then, as we got to about 20 miles from Cardiff, the damn motorway closed down for “works” — possibly related to the upcoming NATO summit and security measures being put in place around town.

I tried to follow the diversion signs but ended up taking the wrong exit from the roundabout and got stuck going north on A449. Unlike our old friend A360 this was a very nice modern highway, but there’s no place to turn around for 10 miles… So it takes a while to get back and try again.

Following the diversion signs we ended up back on M4 but eastbound, back towards London. Argh! We stopped at the next services centre for another break and to regroup.

Google Maps just kept routing us to the closed section of M4 so was of limited help. I called the hotel in Cardiff to ask for a recommended alternate route, but they knew nothing about the closure. I called Europcar but they couldn’t give me anything useful either. Finally, a nice lady at the Costa coffee place overheard our dilemma and offered a route through Newport which would take us around the closure and pick up M4 again. Thanks Becky!

Unfortunately I made a wrong turn and picked up M4 too early, right back at the closure and diversion… And ended up going north on A449 again. We stopped at the first exit to recheck the maps and determined that if we headed back south to the barista’s recommended route and kept going through Newport correctly it would work… But we had to go the 10 miles to the turnaround first, which was very frustrating. Back on the alternate route, another A highway, we entered ROUNDABOUT HELL.

I’m still having nightmares of the Google Maps voice calling out “in 800 feet, at the roundabout, take the second exit to go straight ahead”. Every … fricking … intersection. The alternate route eventually turned out we think to be the recommended diversion route — there were yellow signs with a black circle and a narrow pointing which way to go which lined up with our route and we stuck with that until we returned to the blessed, blessed M4. Finally, we got into Cardiff and Google Maps was relatively sane again leading us to the hotel. We arrived before midnight, but not by much.


Lesson learned: When using your satnav in Britain, research your route first. You can’t tell whether an A road will be comfortable or horrible unless you check it on Wikipedia or something. Gah!

A well deserved post-drive treat.
A well deserved post-drive treat.

Apple Doesn’t Want Developers?

Ok, now this is just confusing:

By now, most of you have probably seen or heard about Apple iPhone SDK 4.0 and a little hidden gem in their freshly minted 4.0 Terms of Services, notably 3.3.1:

“Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited.”

It’s clear that products like Titanium, Unity3D, Ansca, MonoTouch and others are now a bit in question for iPhone 4.0+ with this language. We’re all trying to get our heads around what this means and trying to reach out to Apple to get clarification.

I called up Apple to ask for details about how this affects Appcelerator & Unity3d as particular examples that I’d been planning to use for games and utilities, and whether I can get a refund for my program fees if I were to not accept the updated agreement.

The rep was very pleasant and polite and took my information to escalate the question; the CSRs of course don’t have immediate answers for this. I was told someone should get back to me within 24 hours.

If y’all also call, please above all be polite — customer service reps are real people too, and they didn’t write the agreement! Be polite, accept being on hold gracefully, let them know how this would affect your future development plans, and be specific about what questions you’re asking and what answers you need.