im in ur javascript, decoding ur webm

Been a while since I posted; working on various bits and bobs.

Most exciting at the moment: initial work on pure-JavaScript WebM playback support for ogv.js (soon to become codec.js!)

This is currently very incomplete; video doesn’t come out right and audio’s not hooked up yet, and most notably you only get a few frames before it craps out. :)

Update: Got audio working after some sleep, and playback is more thorough. Yay! Still no seeking yet, and playback may not always reach completion.

I’ve seen a couple other attempts to build WebM decoders in JS using emscripten (eg Route9), but nobody’s gotten them hooked up to a full-blown player with audio yet. With a little more work on the decoder side, I should be able to leverage all of my investment in ogv.js’s player logic.

webm-in-progresslive preview of current experimental build

How’s it work?

I’m using nestegg for the WebM container parsing; currently this is giving me some impedence mismatch with the first-generation ogv.js code due to different i/o models.

ogv.js’s C-side Ogg container parsing was set up with entirely asynchronous i/o due to the needs of using async XHR to fetch data over the network. Nestegg, however, uses synchronous i/o callbacks in some places, and I really don’t want to jump through all the hoops to do WebM cue seeking without using the library. (Already did that with Ogg and it was horrid!)

I may be able to adapt the ogv.js C-side code to using synchronous i/o callbacks and let them get mapped to async i/o on the JS side, thanks to some fancy features in emscripten — either Asyncify or the newer emterpreter mode for parts of the high-level code while keeping the low-level decoder in pure asm.js. Need to do a little more research… :)

Possibilities

Performance: speed on WebM/VP8 decoding won’t be as good as the Ogg/Theora decoder for now so may not be suitable for mobile devices, but should work fine on many laptop/desktop machines.

VP9: untested so far, but should work… though again, likely to be still slower than VP8.

Threading: currently threading is disabled in the build. There’s work ongoing at Mozilla to support pthreads for emscripten programs, using a new SharedArrayBuffer interface to share memory between Web Worker threads. Might experiment with that later as libvpx has some threading options (may only be useful for VP9 multi-slice decoding for now) but it’ll be a while before most browsers support it. However this might be a good use case for asking Microsoft and Apple to add support. :)

SIMD: there’s also ongoing work on SIMD instruction abstraction in JavaScript, with Mozilla and Microsoft partnering. Will have to test with autovectorization in the compiler enabled and see if it makes any difference to decode speed, but don’t know how complete the support in Firefox and Edge is etc.

ogv.js MediaWiki integration updates

Over the last few weekends I’ve continued to poke at ogv.js, both the core library and the experimental MediaWiki integration. It’s getting pretty close to merge-ready!

Recent improvements to ogv.js player (gerrit changeset):

  • Audio no longer super-choppy in background tabs
  • ‘ended’ is no longer unreasonably delayed
  • various code cleanup
  • ogvjs-version.js with build timestamp available for use as a cache-buster helper

Fixes to the MediaWiki TimedMediaHandler desktop player integration (gerrit changeset):

  • Post-playback behavior is now the same as when using native playback
  • Various code cleanup

Fixes to the MediaWiki MobileFrontend mobile player integration (gerrit changeset):

  • Autoplay now working with native playback in Chrome and Firefox
  • Updated to work with current MobileFrontend (internal API changes)
  • Mobile media overlay now directly inherits from the MobileFrontend photo overlay class instead of duplicating it
  • Slow-CPU check is now applied on mobile player — this gets ogv.js video at 160p working on an old iPhone 4S running iOS 7! Fast A7-based iPhones/iPads still get 360p.

While we’re at it, Microsoft is opening up a public ‘suggestion box’ for Internet Explorer — folks might want to put in their votes for native Ogg Vorbis/Theora and WebM playback.

Flame on! Trying out Firefox OS at 2.1…

Ever since I heard about Mozilla’s ‘Boot2Gecko‘ project a few years back I was very excited about the eventual possibility of Firefox-powered phones running a truly free operating system, with apps provided through the open web  instead of platform-lock-in walled gardens.

It’s been a long journey though, and often a painful one. Early versions of Firefox OS were pretty rough, it was hard to get phones that weren’t severely underpowered, and actually upgrading to the latest versions on a release phone was….. often not really possible.

So I finally gave in and picked up the Flame, which is the officially recommended Firefox OS reference device. Current builds are actually, like, published for it!

I immediately flashed the device to the current base image (v180, with a low-level ‘Gonk’ layer based on Android 4.4’s low-level Linux layers) and updated to the almost-ready-for-release Firefox OS 2.1.

Version 2.1 finally does away with the old crappy browser app and treats web site browsing on the same level as installed ‘apps’. Graphics are pretty smooth, using hardware compositing, and in general it’s a HUGE improvement over 1.x.

The Flame

Hardware notes

    • The Flame is meant to be representative of the next generation of Firefox OS release phones which are targeting developing markets, so it’s not as fancy as the latest Android or iOS devices.
      • The screen is only 1.5x density, versus 3x on my Nexus 5. But it’s still a big improvement over the older 1x 320×480 devices.
      • Decent 1GB RAM — can be configured lower to simulate lower-end devices, which I have not attempted. Eek!
      • There’s a limited amount of internal storage, and a micro-SD card slot where you’re expected to store additional files such as media. I only had a 4GB card handy from an old phone so I’m using that for now, but will replace it with a 32GB card later.
    • the Flame has 2 SIM slots, both full-size. This meant I needed a micro-SIM-to-fullsize-SIM adapter to get my main phone line running on the Flame. The micro-SIM kept popping out of the adapter while trying to insert it, but I eventually got it in intact and it’s working fine. (T-Mobile US, HSDPA speeds. No LTE support on the Flame.) Conveniently the adapter kit also included the necessary adapter to move my backup/testing phone line from my iPhone 5s (nano-SIM) to the Nexus 5. Why can’t we all just use the same damn size SIM?

The camera seems kinda awful; video framerate is bad. Not sure if this is a software bug or a hardware limitation but it’s a bit of a bummer.

Back to the web: de-appifying

The most common apps I use on my Nexus 5 are:

    • Gmail
    • Google Maps
    • Facebook
    • Twitter
    • Feedly
    • Kindle
    • Amazon Music

Wikipedia

These are all available on the web, but with some caveats and limitations:

  • Gmail shows in a really crappy old-school mobile web interface instead of the nice modern HTML5 one you get on an Android or iOS device. I can’t seem to use it for multiple accounts either, which makes it a non-starter since I have both personal (gmail) and work (gapps) accounts. I’ve been using the Firefox OS built-in Email app instead for now, which seems to work better than in old versions but isn’t really optimized for my ‘archive everything’ workflow.
  • Google Maps shows the web interface, which is kinda ugly but seems to work including geolocation and transit directions. YAYYY
  • Facebook web seems pretty decent at least for reading, but I don’t get notifications of replies and have to check manually.
  • Twitter web seems pretty good, though the pull-to-refresh is a little flaky and again no notifications.
  • Feedly’s web interface is designed for desktop screens and doesn’t scale down property to a smartphone screen. BOOO
  • Kindle Cloud Reader actually runs — it downloads and views books and everything. But again, it’s designed for desktop and tablet screens and the UI doesn’t scale down. You can only see the top-left corner of the page and can’t actually read anything. BOOOOO
  • Amazon Cloud Player for online-stored music….. amazingly this works, but the interface is desktop-oriented and distinctly NOT mobile friendly. (It also prompts for Adobe Flash, but doesn’t seem to require it for playback.) However since playback stops when you switch away from the app, it’s kind of a bummer to use. BOOOO
  • We have a Firefox OS Wikipedia reader app based on our old PhoneGap app — it works fine, but hasn’t been maintained much and needs an overhaul. Meanwhile our mobile web site also works pretty well on Firefox os, and now supports editing and all kinds of cute stuff. YAYYYY

Now, some things I can’t get at all:

  • Uber
  • Skype
  • Walgreens

:(

  • There’s really nothing in Uber that needs to be an app as a customer — they could just as easily have a web app with all the capabilities of looking up, calling a car, watching the map, etc. I can’t even successfully log in to their web interface for viewing my past rides, but if I could there’s no way to call a cab there.
  • I occasionally use Skype, mainly when XBox Live’s chat system breaks. *cough* Microsoft owns them both *cough*. That’s all native apps and has no web option.
  • The Walgreens app on iOS/Android lets you scan the barcode on your medication to schedule a refill, it’s pretty handy! Their web site has no equivalent that I can find… but I can work around it by renewing via email notification instead.

So I’ll be carrying the Flame around as my main phone line for at least a bit, but I’m gonna keep the Nexus 5 around for a few things.

We’ll see how long it takes before I switch the main line back to Android or if I stick with it. :D

OgvKit: native Ogg Vorbis/Theora playing on iOS

In addition to my in-browser ogv.js media player, I’ve got an OgvKit framework in progress for native iPhone and iPad apps, which I hope to integrate into Wikipedia’s new iOS app somewhere down the line. I took a little Independence Day holiday time and made a bunch of improvements from where I last left it a few months ago:

(If the video is sideways, sorry — it’s still “processing” as of this writing.)

  • Color conversion is OpenGL ES-accelerated, cutting CPU usage in half when playing video.
  • Audio output actually works, more or less in sync.
  • Framework now packaged as a Cocoa Touch Static Library project

Shiny new A7-based 64-bit devices play 480p and even 720p happily (iPhone 5S, iPad Air) but still struggle with some 1080p originals. My oldest devices like an iPhone 3Gs and iPod Touch 4th-gen can’t go higher than 160p or so at present — still significantly faster than the JavaScript ogv.js version which can’t handle video on those devices at all.

Future work for another weekend:

  • It turns out the xiph.org git mirrors of theora etc are not being maintained; will switch fetching of library source to SVN.
  • Improved controls, seeking, fullscreen
  • Cache downloaded data on disk instead of RAM!
  • Move more of the player code from the demo into the library so it can be reused…
  • Test standalone packaging of the library so it can be dropped in to other projects easily
  • Try to get assembly in libtheora to build, and/or replace the ARM assembly code with C SIMD intrinsics
  • Find and test on an armv7s device (iPhone 5, 5C or iPad 4th-generation)

LG G Watch first look

At the Google I/O conference this week they handed out Android Wear watches to attendees; I got the LG G Watch and have been gleefully wearing it for about a day.

The good:

  • There’s no annoying branding on the watch face, and no side buttons to get caught on things.
  • Actual pairing is pretty straightforward using the Android Wear app (once you get it installed…)
  • Showing notifications from my paired Android phone “just works”: texts, Facebook replies, “time for next meeting” pings, etc. It’s also easy to configure it to disable buzzing/pinging on the phone when the watch is active.
  • If you are brave enough to turn on Gmail notifications, you can easily archive a mail from your watch and never have to read it! Or you can swipe away the notification and read it later, if it’s like IMPORTANT or something.
  • Gestures for control are relatively simple.
  • The usual suspects in voice recognition like setting reminders and alarms work, as with Google Now or Glass.
  • If a voice command doesn’t match anything, it does a Google search. Some specific kinds of queries will give results from their knowledge graph, but you can easily end up with generic search results … which inevitably include Wikipedia. :D If you want to actually read a page, it doesn’t try to force it onto the tiny watch screen — it opens up the browser on your phone.
  • The always-on dim display mode looks pretty good indoors or in the shade.

The bad:

  • The screen is nearly illegible in direct sunlight, even pumped up to the max brightness.
  • The G Watch does feel a bit clunky — it’s just kinda big for a watch. But really, it doesn’t feel any worse than my Casio calculator watch did when I was 12. ;)
  • Had to read the directions to see how to turn it on (attach it to the charging cradle and it turns on automatically).
  • Setup currently requires opting in to some prerelease versions of a few Android packages. This presumably will be improved shortly!
  • Voice recognition is a bit spotty. I do wonder what the NSA thinks of my reminder to “resell my medications” (that was “refill”, silly Google!)
  • I wish there were more options for Gmail notifications, namely “mute” and “report spam”.

The ugly:

  • The 280×280 screen resolution looks rather blocky compared to today’s high-end circa-5″ 1080p phones. This is probably a tradeoff for battery life — the watch is already thicker than I’d like, and I wouldn’t want them to have to make the battery huger to last through a day!
  • I had some trouble with the device losing connectivity a couple of times; resetting BlueTooth off and on on the phone seemed to resolve it.

Still have to try:

  • Phone answering — I don’t make or receive a lot of voice calls so haven’t actually tried this yet. Not even sure if it just tells me to grab my phone or if it does some magic watch-speakerphone thing. Who knows? Time will tell.
  • App development — there seem to be two ways to go; either enhanced notifications, or native apps that run on the watch (like the little compass app or the pedometer). Haven’t tried yet, but downloaded the SDK…

iPhone size speculation

So there’s wide speculation out there that a larger-screened iPhone is coming to compete with the circa-5-inch generation of flagship Android phones.

But how would Apple actually engineer such a thing, and what would it mean for web and app developers?

There are a few possibilities to my mind, with various trade offs.

First, note that the iPhone 5 family has a 4″ 640×1136 screen, about 326dpi and using a 2x scale between UI points (aka “CSS pixels”). Software for iPhone had long assumed a 320×480 1x or 640×960 2x display, and when the new screen size was introduced, old apps were accommodated by showing them with black bars to simulate the older screen size.

The simplest solution would be to scale up the iPhone 5c/s design and screen by the same ratio as the iPad mini and iPad Air — this gives a 4.8 inch “iPhone 6″ that would have the same pixel density as the full-size Retina iPad. Developers would see the 4″ and 4.8″ devices as equivalent, with no changes needed to code or graphics. But, it might not provide the feeling of additional screen space because on screen elements would become larger to fit. The resolution, while still a respectable 263dpi or so would also fall noticeably short of the current crop of 1080p Android phones at 440+dpi.

Another possibility is to stick to the 326dpi density and change the screen’s pixel dimensions, say to 1280×720 at 4.5” or maybe a little higher. This would give more onscreen space, requiring app developers to ensure they handle the different dimensions. Older apps without AutoLayout might be handled by a black screen border like iPhone apps running on an iPad, or they might scale up and be a little blurry.

Some might scoff at such a change, but iOS SDK updates have made it easier to handle varying screen sizes, from iOS 6’s introduction of AutoLayout to iOS 8’s storyboard unification and mysterious “variable sized iPhone simulator”… The idea that Apple might spring a new form factor at us in fall 2014 is not any crazier than when they sprung the 4” iPhone 5 on us…

Still another question is whether Apple will follow the dpi race that they started with the iPhone 4’s “retina” screen… Would they make a 1080p phone in the 4.5-5″ range using a 3x display scale to match Android? There’s nothing in the iOS SDK that hints that way to me, but it’s plausible technically.

A 3x density scale might handle back-compatibility by rendering to a 2x frame buffer and transparently scaling up at some cost of blurriness, while newer apps use @3x artwork and render natively at proper scale.

This remains to be seen…

ogv.js sound and video now working on iOS

Thanks to some audio-related fixes and improvements from a contributor, I’ve started doing some new work on ogv.js, the emscripten-powered JavaScript Ogg Theora/Vorbis decoder.

In addition to Internet Explorer 10/11 (via Maik’s Flash shim), I now have audio working on iOS — and smaller video sizes actually play pretty decently on a current iPhone 5s as well!

See demo:

Try it out on your computer or phone!

Older iOS 7 devices and the last generation of iPod Touch are just too slow to play video reliably but still play audio just fine. The latest 64-bit CPU is pretty powerful though, and could probably handle slightly larger transcodes than 160p too.