So I fixed a really annoying bug in my video player program where it sometimes just hung waiting on network input.
I thought it was going to be a really hairy multi-threaded concurrency issue, as I was building a blocking i/o interface on top of a non-blocking URL-fetching layer in order to reuse existing libraries.
I scoured my code looking for incorrect ordering, bad locking, or a problem with my semaphores…. Bit by bit the code was instrumented, logged, inspected, cleared, cleaned, and set back in place better than it was before. But the bug remained, untamed — nay untouched — by my onslaught.
At last I found the part of the code the bug was hiding: if the input buffer ran dry and the last network fetch had already completed, the i/o thread would die and the blocking request would time out.
But why was it broken? Because… um… I forgot to call the function that sends a follow-up request for more data.
I’ve been cleaning up some of my old test code for running Ogg media on iOS, adding WebM support and turning it into OGVKit, a (soon-to-be) reusable library that we can use to finally add video and audio playback to our Wikipedia iPhone app.
Of course decoding VP8 or Theora video on the CPU is going to be more expensive in terms of energy usage than decoding H.264 in dedicated silicon… but how much more?
The iOS 9 beta SDK supports enhanced energy monitoring in Xcode 7 beta… let’s try it out! The diagnostic detail screen looks like so:
Whoa! That’s a little overwhelming. What’s actually going on here?
First, what’s going on here
I’ve got my OGVKit demo app playing this video “Curiosity’s Seven Minutes of Terror” found on Wikimedia Commons, on two devices running iOS 9 beta: an iPod Touch (the lowest-end currently sold iDevice) and an iPad Air (one generation behind the highest-end currently sold iDevice).
The iPod Touch is playing a modest 360p WebM transcode, while the iPad Air is playing a higher-resolution 720p WebM transcode with its beefier 64-bit CPU:
First look: the cost of networking
At first, the energy usage looks pretty high:
This however is because in addition to media playback we’re buffering umpty-ump megabytes over HTTPS over wifi — as fast as a 150 Mbps cable connection will allow.
Once the download completes, the CPU usage from SSL decoding goes down, the wifi reduces its power consumption, and our energy usage relatively flattens.
Now what’s the spot-meter look like?
Pretty cool, right!?
See approximate reported energy usage levels for all transcode formats (Ogg Theora and WebM at various resolutions) if you like! Ogg Theora is a little faster to decode but WebM looks significantly better at the bitrates we use.
Ok but how’s that compare to native H.264 playback?
Good question. I’m about to try it and find out.
Ok here’s what we got:
The native AVPlayer downloads smaller chunks more slowly, but similarly shows higher CPU and energy usage during download. Once playing only, reported CPU usage dives to a percent or two and the reported energy impact is “Zero”.
Now, I’m not sure I believe “Zero”… 😉
I suppose I’ll have to rig up some kind of ‘run until the battery dies’ test to compare how reasonable this looks for non-trivial playback times… but the ‘Low’ reportage for WebM at reasonable resolutions makes me happier than ‘Very High’ would have!
I’ve been passing the last few days feverishly working on audio/video stuff, cause it’s been driving me nuts that it’s not quite in working shape.
TL;DR: Major fixes in the works for Android, Safari (iOS and Mac), and IE/Edge (Windows). Need testers and patch reviewers.
ogv.js for Safari/IE/Edge
I’ll want to update it to work with Video.js later, but I’d love to get this version reviewed and deployed in the meantime.
Please head over to https://ogvjs-testing.wmflabs.org/ in Safari 6.1+ or IE 10+ (or ‘Project Spartan’ on Windows 10 preview) and try it out! Particularly interested in cases where it doesn’t work or messes up.
However these get really bad compression ratios, so to keep bandwidth down similar to the 360p Ogg and WebM versions I had to reduce quality and resolution significantly. Hold an iPhone at arm’s length and it’s maybe ok, but zoom full-screen on your iPad and you’ll hate the giant blurry pixels!
This should also provide a working basic audio/video experience in our Wikipedia iOS app, until such time as we integrate Ogg or WebM decoding natively into the app.
Note that it seems tricky to bulk-run new transcodes on old files with TimedMediaHandler. I assume there’s a convenient way to do it that I just haven’t found in the extension maint scripts…
In progress: mobile video fixes
Audio has worked on Android for a while — the .ogg files show up in native <audio> elements and Just Work.
But video has been often broken, with TimedMediaHandler’s “popup transforms” reducing most video embeds into a thumbnail and a link to the original file — which might play if WebM (not if Ogg Theora) but it might also be a 1080p original which you don’t want to pull down on 3G! And neither audio nor video has worked on iOS.
This patch adds a simple mobile target for TMH, which fixes the popup transforms to look better and actually work by loading up an embedded-size player with the appropriately playable transcodes (WebM, Ogg, and the MJPEG last-ditch fallback).
ogv.js is used if available and necessary, for instance in iOS Safari when the CPU is fast enough. (Known to work only on 64-bit models.)
Future: codec.js and WebM and OGVKit
For the future, I’m also working on extending ogv.js to support WebM for better quality (especially in high-motion scenes) — once that stabilizes I’ll rename the combined package codec.js. Performance of WebM is not yet good enough to deploy, and some features like seeking are still missing, but breaking out the codec modules means I can develop the codecs in parallel and keep the high-level player logic in common.
Browser infrastructure improvements like SIMD, threading, and more GPU access should continue to make WebM decoding faster in the future as well.
I’d also like to finish up my OGVKit package for iOS, so we can embed a basic audio/video player at full quality into the Wikipedia iOS app. This needs some more cleanup work still.
This is currently very incomplete; video doesn’t come out right and audio’s not hooked up yet, and most notably you only get a few frames before it craps out.
Update: Got audio working after some sleep, and playback is more thorough. Yay! Still no seeking yet, and playback may not always reach completion.
I’ve seen a couple other attempts to build WebM decoders in JS using emscripten (eg Route9), but nobody’s gotten them hooked up to a full-blown player with audio yet. With a little more work on the decoder side, I should be able to leverage all of my investment in ogv.js’s player logic.
I’m using nestegg for the WebM container parsing; currently this is giving me some impedence mismatch with the first-generation ogv.js code due to different i/o models.
ogv.js’s C-side Ogg container parsing was set up with entirely asynchronous i/o due to the needs of using async XHR to fetch data over the network. Nestegg, however, uses synchronous i/o callbacks in some places, and I really don’t want to jump through all the hoops to do WebM cue seeking without using the library. (Already did that with Ogg and it was horrid!)
I may be able to adapt the ogv.js C-side code to using synchronous i/o callbacks and let them get mapped to async i/o on the JS side, thanks to some fancy features in emscripten — either Asyncify or the newer emterpreter mode for parts of the high-level code while keeping the low-level decoder in pure asm.js. Need to do a little more research…
Performance: speed on WebM/VP8 decoding won’t be as good as the Ogg/Theora decoder for now so may not be suitable for mobile devices, but should work fine on many laptop/desktop machines.
VP9: untested so far, but should work… though again, likely to be still slower than VP8.
Threading: currently threading is disabled in the build. There’s work ongoing at Mozilla to support pthreads for emscripten programs, using a new SharedArrayBuffer interface to share memory between Web Worker threads. Might experiment with that later as libvpx has some threading options (may only be useful for VP9 multi-slice decoding for now) but it’ll be a while before most browsers support it. However this might be a good use case for asking Microsoft and Apple to add support.
Ever since I heard about Mozilla’s ‘Boot2Gecko‘ project a few years back I was very excited about the eventual possibility of Firefox-powered phones running a truly free operating system, with apps provided through the open web instead of platform-lock-in walled gardens.
It’s been a long journey though, and often a painful one. Early versions of Firefox OS were pretty rough, it was hard to get phones that weren’t severely underpowered, and actually upgrading to the latest versions on a release phone was….. often not really possible.
So I finally gave in and picked up the Flame, which is the officially recommended Firefox OS reference device. Current builds are actually, like, published for it!
I immediately flashed the device to the current base image (v180, with a low-level ‘Gonk’ layer based on Android 4.4’s low-level Linux layers) and updated to the almost-ready-for-release Firefox OS 2.1.
Version 2.1 finally does away with the old crappy browser app and treats web site browsing on the same level as installed ‘apps’. Graphics are pretty smooth, using hardware compositing, and in general it’s a HUGE improvement over 1.x.
The Flame is meant to be representative of the next generation of Firefox OS release phones which are targeting developing markets, so it’s not as fancy as the latest Android or iOS devices.
The screen is only 1.5x density, versus 3x on my Nexus 5. But it’s still a big improvement over the older 1x 320×480 devices.
Decent 1GB RAM — can be configured lower to simulate lower-end devices, which I have not attempted. Eek!
There’s a limited amount of internal storage, and a micro-SD card slot where you’re expected to store additional files such as media. I only had a 4GB card handy from an old phone so I’m using that for now, but will replace it with a 32GB card later.
the Flame has 2 SIM slots, both full-size. This meant I needed a micro-SIM-to-fullsize-SIM adapter to get my main phone line running on the Flame. The micro-SIM kept popping out of the adapter while trying to insert it, but I eventually got it in intact and it’s working fine. (T-Mobile US, HSDPA speeds. No LTE support on the Flame.) Conveniently the adapter kit also included the necessary adapter to move my backup/testing phone line from my iPhone 5s (nano-SIM) to the Nexus 5. Why can’t we all just use the same damn size SIM?
The camera seems kinda awful; video framerate is bad. Not sure if this is a software bug or a hardware limitation but it’s a bit of a bummer.
Back to the web: de-appifying
The most common apps I use on my Nexus 5 are:
These are all available on the web, but with some caveats and limitations:
Gmail shows in a really crappy old-school mobile web interface instead of the nice modern HTML5 one you get on an Android or iOS device. I can’t seem to use it for multiple accounts either, which makes it a non-starter since I have both personal (gmail) and work (gapps) accounts. I’ve been using the Firefox OS built-in Email app instead for now, which seems to work better than in old versions but isn’t really optimized for my ‘archive everything’ workflow.
Google Maps shows the web interface, which is kinda ugly but seems to work including geolocation and transit directions. YAYYY
Facebook web seems pretty decent at least for reading, but I don’t get notifications of replies and have to check manually.
Twitter web seems pretty good, though the pull-to-refresh is a little flaky and again no notifications.
Feedly’s web interface is designed for desktop screens and doesn’t scale down property to a smartphone screen. BOOO
Kindle Cloud Reader actually runs — it downloads and views books and everything. But again, it’s designed for desktop and tablet screens and the UI doesn’t scale down. You can only see the top-left corner of the page and can’t actually read anything. BOOOOO
Amazon Cloud Player for online-stored music….. amazingly this works, but the interface is desktop-oriented and distinctly NOT mobile friendly. (It also prompts for Adobe Flash, but doesn’t seem to require it for playback.) However since playback stops when you switch away from the app, it’s kind of a bummer to use. BOOOO
We have a Firefox OS Wikipedia reader app based on our old PhoneGap app — it works fine, but hasn’t been maintained much and needs an overhaul. Meanwhile our mobile web site also works pretty well on Firefox os, and now supports editing and all kinds of cute stuff. YAYYYY
Now, some things I can’t get at all:
There’s really nothing in Uber that needs to be an app as a customer — they could just as easily have a web app with all the capabilities of looking up, calling a car, watching the map, etc. I can’t even successfully log in to their web interface for viewing my past rides, but if I could there’s no way to call a cab there.
I occasionally use Skype, mainly when XBox Live’s chat system breaks. *cough* Microsoft owns them both *cough*. That’s all native apps and has no web option.
The Walgreens app on iOS/Android lets you scan the barcode on your medication to schedule a refill, it’s pretty handy! Their web site has no equivalent that I can find… but I can work around it by renewing via email notification instead.
So I’ll be carrying the Flame around as my main phone line for at least a bit, but I’m gonna keep the Nexus 5 around for a few things.
We’ll see how long it takes before I switch the main line back to Android or if I stick with it. 😀
So I’m doing a little post-Wikimania traveling with my wife and my parents. Yesterday we drove from London to Cardiff via Stonehenge. It was… Quite the experience for a first-time driving in the UK.
Part one: Escaping London
Our adventure begins in the London Docklands, where the 72nd World Science Fiction Convention was held at the Excel Centre (loncon3.org). I was able to hire a car at the Europcar branch in the convention center, made it over to our hotel, and we just managed to squeeze our luggage into the back of this Skoda something or other.
Google Maps wanted to route us through the London city center to get out to the M4 motorway, but everyone I asked assured me this was a terrible idea and I should get to the M25 “orbital” highway that circles the city. A13 runs east from the docklands to the M25 and was pretty easy to get to; after some initial confusion getting used to driving on the left and being on the right side of the car I more or less adjusted, and we stopped for a quick lunch at a rest stop (“services centre”) off M25.
Part two: reaching Stonehenge
From there the route to Stonehenge was very simple: go south and west on the M25 orbital until the M3 branches off, then take A303 out to Amesbury and follow the signs to Stonehenge. This route was great; mostly big modern highways, well labeled, in the middle of the day. My main difficulty was adjusting to properly centering the car in the lane when I’m sitting on the “wrong” side of the car.
Part three: English country back road hell
When I planned out the route I didn’t do enough research on how to get back to the main motorway; it looked clear enough on Google Maps and I just turned on navigation on my phone and followed the directions a while.
The phone losing gps signal at first was a bad sign, but in retrospect the route was bad to begin with. We ended up taking A360 sorta northwestward toward the M4 which leads straight to Cardiff. As it turns out, while A303 was mostly a pretty comfortable minor highway, A360 is actually a series of tiny country and village back roads.
Often it narrows to one lane, has no shoulder, squirrels around and makes weird turns, etc. this was a somewhat harrowing experience, especially as signage was nearly nonexistent and I had a poor idea of how far I was from the main highway.
Part four: finding M4
Eventually we reached the entrance to M4… And I missed the exit from the roundabout and ended up on the wrong road. Google Maps rerouted us… Down another country road which eventually took us back to M4, much much later than I had hoped to be on the main road.
Once on M4 we were back in a world of wide lanes, divided highways, good signage, etc. Life was good again. We kept going west, crossing the Severn bridge to Wales. Interestingly this is a toll bridge westbound, but the toll collection is a good few miles past the bridge instead of before it like San Francisco’s bridges.
Part five: diversion hell
Then, as we got to about 20 miles from Cardiff, the damn motorway closed down for “works” — possibly related to the upcoming NATO summit and security measures being put in place around town.
I tried to follow the diversion signs but ended up taking the wrong exit from the roundabout and got stuck going north on A449. Unlike our old friend A360 this was a very nice modern highway, but there’s no place to turn around for 10 miles… So it takes a while to get back and try again.
Following the diversion signs we ended up back on M4 but eastbound, back towards London. Argh! We stopped at the next services centre for another break and to regroup.
Google Maps just kept routing us to the closed section of M4 so was of limited help. I called the hotel in Cardiff to ask for a recommended alternate route, but they knew nothing about the closure. I called Europcar but they couldn’t give me anything useful either. Finally, a nice lady at the Costa coffee place overheard our dilemma and offered a route through Newport which would take us around the closure and pick up M4 again. Thanks Becky!
Unfortunately I made a wrong turn and picked up M4 too early, right back at the closure and diversion… And ended up going north on A449 again. We stopped at the first exit to recheck the maps and determined that if we headed back south to the barista’s recommended route and kept going through Newport correctly it would work… But we had to go the 10 miles to the turnaround first, which was very frustrating. Back on the alternate route, another A highway, we entered ROUNDABOUT HELL.
I’m still having nightmares of the Google Maps voice calling out “in 800 feet, at the roundabout, take the second exit to go straight ahead”. Every … fricking … intersection. The alternate route eventually turned out we think to be the recommended diversion route — there were yellow signs with a black circle and a narrow pointing which way to go which lined up with our route and we stuck with that until we returned to the blessed, blessed M4. Finally, we got into Cardiff and Google Maps was relatively sane again leading us to the hotel. We arrived before midnight, but not by much.
Lesson learned: When using your satnav in Britain, research your route first. You can’t tell whether an A road will be comfortable or horrible unless you check it on Wikipedia or something. Gah!
In addition to my in-browser ogv.js media player, I’ve got an OgvKit framework in progress for native iPhone and iPad apps, which I hope to integrate into Wikipedia’s new iOS app somewhere down the line. I took a little Independence Day holiday time and made a bunch of improvements from where I last left it a few months ago:
(If the video is sideways, sorry — it’s still “processing” as of this writing.)
Color conversion is OpenGL ES-accelerated, cutting CPU usage in half when playing video.
Audio output actually works, more or less in sync.
Framework now packaged as a Cocoa Touch Static Library project
Future work for another weekend:
It turns out the xiph.org git mirrors of theora etc are not being maintained; will switch fetching of library source to SVN.
Improved controls, seeking, fullscreen
Cache downloaded data on disk instead of RAM!
Move more of the player code from the demo into the library so it can be reused…
Test standalone packaging of the library so it can be dropped in to other projects easily
Try to get assembly in libtheora to build, and/or replace the ARM assembly code with C SIMD intrinsics
Find and test on an armv7s device (iPhone 5, 5C or iPad 4th-generation)
At the Google I/O conference this week they handed out Android Wear watches to attendees; I got the LG G Watch and have been gleefully wearing it for about a day.
There’s no annoying branding on the watch face, and no side buttons to get caught on things.
Actual pairing is pretty straightforward using the Android Wear app (once you get it installed…)
Showing notifications from my paired Android phone “just works”: texts, Facebook replies, “time for next meeting” pings, etc. It’s also easy to configure it to disable buzzing/pinging on the phone when the watch is active.
If you are brave enough to turn on Gmail notifications, you can easily archive a mail from your watch and never have to read it! Or you can swipe away the notification and read it later, if it’s like IMPORTANT or something.
Gestures for control are relatively simple.
The usual suspects in voice recognition like setting reminders and alarms work, as with Google Now or Glass.
If a voice command doesn’t match anything, it does a Google search. Some specific kinds of queries will give results from their knowledge graph, but you can easily end up with generic search results … which inevitably include Wikipedia. 😀 If you want to actually read a page, it doesn’t try to force it onto the tiny watch screen — it opens up the browser on your phone.
The always-on dim display mode looks pretty good indoors or in the shade.
The screen is nearly illegible in direct sunlight, even pumped up to the max brightness.
The G Watch does feel a bit clunky — it’s just kinda big for a watch. But really, it doesn’t feel any worse than my Casio calculator watch did when I was 12. 😉
Had to read the directions to see how to turn it on (attach it to the charging cradle and it turns on automatically).
Setup currently requires opting in to some prerelease versions of a few Android packages. This presumably will be improved shortly!
Voice recognition is a bit spotty. I do wonder what the NSA thinks of my reminder to “resell my medications” (that was “refill”, silly Google!)
I wish there were more options for Gmail notifications, namely “mute” and “report spam”.
The 280×280 screen resolution looks rather blocky compared to today’s high-end circa-5″ 1080p phones. This is probably a tradeoff for battery life — the watch is already thicker than I’d like, and I wouldn’t want them to have to make the battery huger to last through a day!
I had some trouble with the device losing connectivity a couple of times; resetting BlueTooth off and on on the phone seemed to resolve it.
Still have to try:
Phone answering — I don’t make or receive a lot of voice calls so haven’t actually tried this yet. Not even sure if it just tells me to grab my phone or if it does some magic watch-speakerphone thing. Who knows? Time will tell.
App development — there seem to be two ways to go; either enhanced notifications, or native apps that run on the watch (like the little compass app or the pedometer). Haven’t tried yet, but downloaded the SDK…