I support marriage equality, and I am deeply troubled by the tone of recent discourse in the Mozilla community.
People should be allowed to be wrong. People should be capable of working together on common goals despite their differences in other areas.
I appreciate that in California us “liberals” and “progressives” have things pretty good and can afford to take on long-neglected issues like equality of legal marriage rights for everybody in a loving, committed relationship regardless of gender or sexuality. I think that’s great, and I’m proud to live in a state that has been willing to take on the issue. It feels like we live at the forefront of history on basic freedoms for all sorts of often hated and misunderstood groups.
But the world isn’t a single-issue voter. The world is BILLIONS OF PEOPLE, most of whom believe things you or I find highly offensive and vice-versa.
If the tables were turned it might easily be atheists like me, or gays like many of my colleagues and friends who are on the wrong end of the stick — as we and they often have been for generations.
I choose instead to reject the stick as a tool and embrace tolerance and forgiveness.
Some may accuse me of false equivalence. All I know is that destroying someone’s career doesn’t feel like a victory.
Now can we all get back to writing software that improves peoples’ lives?
Two major updates to ogv.js Theora/Vorbis video/audio player in the last few weekends: an all-Flash decoder for older IE versions, and WebGL and Stage3D GPU acceleration for color conversion and drawing.
Try the demo and select ‘JS + WebGL‘, ‘Flash’, or even ‘Flash + GPU‘ from the drop-down! (You can also now try playback in the native video element or the old Cortado Java applet for comparison, though Cortado requires adding security exceptions if your browser works with Java at all these days. :P)
Flash / Crossbridge
The JS code output by emscripten requires some modern features like typed arrays, which aren’t available on IE 9 and older… but similar features exist in Flash’s ActionScript 3 virtual machine, and reasonably recent Flash plugins are often available on older IE installations.
A few interesting notes:
- Crossbridge compiler runs much slower than emscripten does, perhaps due to JVM startup costs in some backend tool. Running the configure scripts on the libraries is painfully slowwwwwwwwww! Luckily once they’re built they don’t have to be rebuilt often.
- There are only Mac and Windows builds of Crossbridge available; it may or may not be possible to build on Linux from source. I’ve only tested on Mac so far.
- Flash decoder performance is somewhere on par with Safari and usually a bit better than Internet Explorer’s JS.
- JS’s typed array & ArrayBuffer system doesn’t quite map to the Flash interfaces. There’s a ByteArray which is kind of like a Uint8Array plus a data stream interface, with methods to read/write values of other types into the current position and advance it. There are also Vector types, which have an interface more like the traditional untyped Array but can only contain items of the given type and are more efficiently packed in memory.
GPU acceleration: WebGL
Safari unfortunately doesn’t enable WebGL by default; it can be enabled as a developer option on Mac OS X but requires a device jailbreak on iOS.
However for IE 11, and the general fun of it, I suspected adding GPU acceleration might make sense! YCbCr to RGB conversion and drawing bytes to the canvas with putImageData() are both expensive, especially in IE. The GPU can perform the two operations together, and more importantly can massively parallelize the colorspace conversion.
I snagged the fragment shader from the Broadway H.264 player project’s WebGL accelerated canvas, and with a few tutorials, a little documentation, and a lot of trial and error I got accelerated drawing working in Firefox, Chrome, and Safari with WebGL manually enabled.
Then came time to run it on IE 11… unfortunately it turns out that single-channel luminance or alpha textures aren’t supported on IE 11’s WebGL — you must upload all textures as RGB or RGBA.
Copying data from 1-byte-per-pixel arrays to a 3- or 4-byte-per-pixel array and then uploading that turned out to be horribly slow, especially as IE’s typed array ‘set’ method and copy constructor seem to be hideously slow. It was clear this was not going to work.
I devised a method of uploading the 1-byte-per-pixel arrays as pseudo-RGBA textures of 1/4 their actual width, then unpacking the subpixels from the channels in the fragment shader.
The unpacking is done by adding two more textures: “stripe” textures at the luma and chroma resolutions, which for each pixel have a 100% brightness in the channel for the matching packed subpixel. For each output pixel, we sample both the packed Y, Cb, or Cr texture and the matching-size stripe texture, then multiple the vectors and sum the components to fold down only the relevant channel into a scalar value.
It feels kinda icky, but it seems to run just as fast at least on my good hardware, and works in IE 11 as well as the other browsers.
On Firefox with asm.js-optimized Theora decoding and WebGL-optimized drawing, I can actually watch 720p and some 1080p videos at full speed. Nice! (Of course, Firefox can already use its native decoder which is faster still…)
A few gotchas with WebGL:
- There’s a lot of boilerplate you have to do to get anything running; spend lots of time reading those tutorials that start with a 2d triangle or a rectangle until it makes sense!
- Once you’ve used a canvas element for 2d rendering, you can’t switch it to a WebGL canvas. It’ll just fail silently…
- Creating new textures is a lot slower than uploading new data to an existing texture of the same size.
- Error checking can be expensive because it halts the GPU pipeline; this significantly slows down the rendering in Chrome. Turn it off once code is working…
GPU acceleration: Flash / Stage3D
Once I had things working so nicely with WebGL, the Flash version started to feel left out — couldn’t it get some GPU love too?
Well luckily, Flash has an OpenGL ES-like API as well: Stage3D.
Unluckily, Stage3D and WebGL are gratuitously different in API signatures. Meh, I can work with that.
Really unluckily, Stage3D doesn’t include a runtime shader compiler.
You’re apparently expected to write shaders in a low-level assembly language, AGAL… and manually grab an “AGALMiniAssembler” class and stick it in your code to compile that into AGAL bytecode. What?
Luckily there’s also a glsl2agal converter, so I was able to avoid rewriting the shaders from the WebGL version in AGAL manually. Yay! This required some additional manual hoop-jumping to make sure variables mapped to the right registers, but the glsl2agal compiler makes the mappings available in its output so that wasn’t too bad.
Some gotchas with Stage3D:
- The AS3 documentation pages for Stage3D classes don’t show all the class members in Firefox. No, really, I had to read those pages in Chrome. WTF?
- Texture dimensions must be powers of 2, so I have to copy rows of bytes from the Crossbridge C heap into a temporary array of the right size before uploading to the GPU. Luckily copying byte arrays is much faster in Flash than in IE’s JS!
- As with the 2d BitmapData interface, Flash prefers BGR over RGB. Just had to flip the order of bytes in the stripe texture generation.
- Certain classes of errors will crash Flash and IE together. Nice!
- glsl2agal compiler forced my texture sampling to linear interpolation with wrapping; I had to do a string replace on the generated AGAL source to set it to nearest neighbor & clamped.
- There doesn’t appear to be a simple way to fix the Stage3D backing buffer size and scale it up to fit the window, at least in the modes I’m using it. I’m instead handling resizing by setting the backing buffer to the size of the stage, and just letting the texture render larger. This unfortunately uses nearest-neighbor for the scaling because I had to disable linear sampling to do the channel-packing trick.
- On my old Atom-based tablet, Stage3D drawing is actually slower than software conversion and rendering. D’oh! Other machines seem faster, and about on par with WebGL.
I’ll do a little more performance tweaking, but it’s starting to look like it’s time to clean it up and try integrating with our media playback tools on MediaWiki… Wheeeee!
This works both with Web Audio in Firefox, Chrome, and Safari and with the Flash audio shim for IE; basically we have to keep track of the audio playback position and match up decoding frames with that. Took a little poking in the ActionScript code but it’s now working!
This brings us much closer to being able to integrate ogv.js as a fallback video player for Wikimedia on IE 10/11 and Safari 6/7. Thanks to the guys hanging out in #xiph channel who encouraged me to keep poking at this! 😀
Additionally there’s now an override selector for the video size, so you can try decoding larger than 360p versions, or switch a slow machine down to the little 160p versions.
I’ve also started investigating an all-Flash version using Adobe’s Crossbridge, which if it works would be a suitable replacement for the Cortado Java applet on old browsers (think of all those IE 6/7/8/9 systems out there!). I seem to be able to build the ogg libraries but haven’t gone beyond that yet… will be interesting to poke at.
In addition to Internet Explorer 10/11 (via Maik’s Flash shim), I now have audio working on iOS — and smaller video sizes actually play pretty decently on a current iPhone 5s as well!
Try it out on your computer or phone!
Older iOS 7 devices and the last generation of iPod Touch are just too slow to play video reliably but still play audio just fine. The latest 64-bit CPU is pretty powerful though, and could probably handle slightly larger transcodes than 160p too.
Latest demo screencast of Android and iOS new Wikipedia mobile apps in development — now with login and basic editing! I think folks are really going to like what we’ve done once we get these polished up and out in the stores replacing our old mobile app.
If the on-wiki embedded video player doesn’t work for you, try a mirror on YouTube.
Thanks to Android 4.4’s built-in screen recording feature this didn’t require any custom hardware kit to record, but I had to jump through some hoops to edit it in PiTiVi… I’ll write up some notes on-wiki.
Mostly this means Internet Explorer and Safari — Chrome and Firefox handle the files natively. However Internet Explorer was limited by the lack of support for the Web Audio API, so could not play any sound. I’d hypothesized that a Flash shim could be used — Windows 8 ships with the Flash plugin by default and it’s widely installed on Windows 7 — but had no idea where to start.
Open source to the rescue!
One of the old maintainers of the Cortado applet, maikmerten, took an interest. After some brief fixes to get the build scripts working on Ubuntu, he scrounged up a simple ActionScript audio shim, with source and .swf output, and rigged up the ogv.js player to output audio through that if there was no native Web Audio API.
The ActionScript of the Flash shim is pretty straightforward, and it compiles into a nice, approx 1kb .swf file. Luckily, you can rebuild the .swf with the open-source Apache Flex SDK, so it doesn’t even rely on proprietary Flash Builder or anything. We could do with some further cleanup (for instance I don’t think we’re disposing of the Flash plugin when shutting down audio, but that’s easy to fix in a bit…) but the basics are in place. And of course getting proper audio/video sync will be complicated by the shim layer — the current code drives the clock based on the video and has choppy audio anyway, so there’s some ways to go before we reach that problem. 😉
It even works on Windows RT, the limited ARM version of Windows 8 — though the video decoding is much too slow on a first-gen Surface tablet’s Tegra 3 CPU, audio-only files play nicely.
Today is Veterans Day in the US, known as Armistice Day in some other countries. Whatever you call it, it’s the day we remember the sacrifices of soldiers and other military veterans, both present and past.
I’ve never been in a war; I’ve never served in any army; I didn’t even grow up in a military family. But war and its human element have always loomed in the background, both informing the history of the world around me and shaping what comes next.
At least one of my great grandfathers served in World War I, a war now nearly a century old that America barely remembers. But it was this war, the “War to End All Wars”, that brought us the 11/11 Armistice Day.
My grandfathers both volunteered during World War II back in the 1940s — the last time the US was on a “total war” footing. My father’s father served in Europe as an Army supply sergeant, and my mother’s father, too young for the Navy, joined the Merchant Marine and helped with trans-Atlantic shipping. They worked hard for their country, and the work they did on the supply lines helped keep front-line soldiers alive and fighting against the Nazis.
As we get closer to the present, America’s wars have gotten smaller, and fewer people have been directly involved on “our” side. My father was in university during Vietnam, so avoided the draft and instead had great learning and job opportunities here at home. War became something remote and theoretical to our family.
When I was a child, Vietnam was over, the draft had been abandoned, but the Cold War was still alive and well; we were more worried about mutual nuclear annihilation with the Soviet Union than conventional war. It almost seemed an anachronism that we had two Marine Corps air bases nearby (MCAS Tustin and MCAS El Toro, which put on a wonderful air show every year for the kids). Tanks and missiles and machine guns and attack helicopters were “cool” things you mostly saw in movies and video games.
But real wars kept happening, even if they weren’t quite as globe-spanning, and real people were still living and dying in them.
Desert Shield / Desert Storm brought active war into focus for me when I was about 12. I didn’t know anybody directly in the military, but I knew they were real people — a lot of people and equipment shipped out from the local bases, and the news would report on casualties from the local area.
One of my most vivid memories from Desert Storm was reading a newspaper account of a friendly fire incident in which a member of a tank crew was decapitated. I’ve seen far more gruesome things simulated in movies and for real in pictures on the internet (unfortunately), but the shock of a 12-year old reading about a soldier suddenly finding himself holding his crewmate’s lifeless head will never quite fade.
“My” generation’s war didn’t come until 9/11 sparked a US invasion of Afghanistan, followed later by another Iraq invasion. There was no longer a draft, but like my father during Vietnam I was old enough I could have served, but chose instead to stick with university and a career.
I don’t like the idea of war. I hate the idea of people being hurt, displaced, killed, or losing loved ones. But war is a real thing that’s part of the human condition, and for better or worse we have to have people who get involved in them to try to bring the fighting to a close.
Anyway, I’m not really sure I had a point. But please, if you have the day off today, spend some time thinking about the people who go off to war, whether deliberately as soldiers or with no choice as noncombatants, who never come back. And the ones that do come back, often don’t come back the same.
Think about this when you make decisions about your life.
Well, I did some more hacking on it this weekend:
- Color output? Check.
- Streaming input? Check. (But no seeking yet, and buffering is aggressive on some browsers.)
- Sound? Check. (But it’s not synced, choppy, and usually the wrong sample rate. And no IE or iOS support yet.)
- Pretty interface to select any file from Wikimedia Commons’ Media of the Day archive? Check.
- Laid some groundwork for separating the codec to a background Worker thread (not yet implemented)
- See the readme for more info.
Feel free to try building or hacking the source for fun.
Larger files run… veerrryyyy sslllooww on my test iPod Touches, but this certainly seems fast enough on desktop to one day replace our old Java fallback for Safari and newer IE…
How it works
Only a couple of tiny tweaks to the libraries are needed to make them build; I started with build scripts for just the audio codecs from this project, added in libtheora, and started adapting parts of one of the Theora data dump examples.
Finish up the YCbCr->RGB conversion, add audio decoding & output, and some kind of sync and seeking, and …… this could replace the old Java Cortado app we use as a fallback player on Wikipedia for browsers that don’t run WebM or Theora.
Web Workers could be used to push decoding to a background thread, depending on whether overhead is problematic.
Crazy idea: provide an HTML5 <video>-style DOM interface, integrate into TimedMediaHandler as a drop-in replacement
- Audio sync may be difficult to achieve.
- Audio output APIs — need to confirm what’s consistently available.
- Performance is surprisingly good on a desktop; I have no doubt this will be sufficient for playback in Safari and IE if audio & sync can be managed.
- Performance is not so good on my test iPod Touches; it might be fun to tune and optimize but I would expect to get much better results from a native app on iOS.
- There doesn’t seem to be a good universal way to do progressive data reads from an XMLHttpRequest; it may be necessary to buffer portions of the file by running multiple requests for subranges of the file, which is NOT pretty.
- IE 10 doesn’t support ArrayBuffer.slice(). Currently this prevents the demo from running, but it’s not actually needed.