Category: android

Daydream View phone VR headset

I somehow ended up with a $100 credit at the Google Store, and decided to splurge on something I didn’t need but wanted — the Daydream View phone VR headset.

This is basically one step up from the old Google Cardboard viewer where you put your phone in a lens/goggles getup, but it actually straps to your head so you don’t have to hold it to your face manually. It’s also got a wireless controller instead of just a single clicky button, so you get a basic pointer within the VR world with selection & home buttons and even a little touchpad, allowing a fair range of controls within VR apps.

Using the controller, the Daydream launcher interface also provides an in-VR UI for browsing Google Play Store and purchasing new games/apps. You can even broadcast a 2d view to a ChromeCast device to share your fun with the rest of the room, though there’s no apparent way to save a recording.

The hardware is still pretty limiting compared to what the PC VR headsets like the Rift and Vive can do — there’s no positional tracking for either head tracking or the controller, so you’re limited to rotation. This means things in the VR world don’t move properly as you move your head around, and you can only use the controller to point, not to pick up VR objects in 3d space. And it’s still running on a phone (Pixel 2 in my case) so you’re not going to get the richest graphics — most apps I’ve tried so far have limited, cartoony graphics.

Still it’s pretty fun to play with; I’ll probably play through a few of these games and then put it away mostly forever, unless I figure out some kind of Wikipedia-related VR project to work on on the side. :)

Alliance for Open Media code drop & more hardware partners

Very exciting! The new video codec so far is mostly based on Google’s in-development VP10 (next gen of the VP8/VP9 used in WebM) but is being co-developed and supported with a number of other orgs.

  • CPU/GPU/SoC makers: Intel, AMD, ARM, NVidia
  • OS & machine makers: Google, Microsoft, Cisco
  • Browser makers: Mozilla, Google, Microsoft
  • Content farms: Netflix, Google (YouTube)

Microsoft is also actively working on VP8/VP9 support for Windows 10, with some limited compatibility in preview releases.

As always, Apple remains conspicuously absent. :(

Like the earlier VP8/VP9 the patent licenses are open and don’t have the kind of weird clauses that have tripped up MPEGLA’s H.264 and HEVC/H.265 in some quarters. (*cough* Linux *cough* Wikipedia)

Totally trying to figure out how we can get involved at this stage; making sure I can build the codec in my iOS app and JavaScript shim environments will be a great start!

Using Web Worker threading in ogv.js for smoother playback

I’ve been cleaning up MediaWiki’s “TimedMediaHandler” extension in preparation for merging integration with my ogv.js JavaScript media player for Safari and IE/Edge when no native WebM or Ogg playback is available. It’s coming along well, but one thing continued to worry me about performance: when things worked it was great, but if the video decode was too slow, it could make your browser very sluggish.

In addition to simply selecting too high a resolution for your CPU, this can strike if you accidentally open a debug console during playback (Safari, IE) or more seriously if you’re on an obscure platform that doesn’t have a JavaScript JIT compiler… or using an app’s embedded browser on your iPhone.

Or simply if the integration code’s automatic benchmark overestimated your browser speed, running too much decoding just made everything crap.

Luckily, the HTML5 web platform has a solution — Web Workers.

Workers provide a limited ability to do multithreading in JavaScript, which means the decoder thread can take as long as it wants — the main browser thread can keep responding to user input in the meantime.

The limitation is that scripts running in a Worker have no direct access to your code or data running in the web page’s main thread — you can communicate only by sending messages with ‘raw’ data types. Folks working with lots of DOM browser nodes thus can’t get much benefit, but for buffer-driven compute tasks like media decoding it’s perfect!

Threading comms overhead

My first attempt was to take the existing decoder class (an emscripten module containing the Ogg demuxer, Theora video decoder, and Vorbis audio decoder, etc) and run it in the worker thread, with a proxy object sending data and updated object properties back and forth.

This required a little refactoring to make the decoder interfaces asynchronous, taking callbacks instead of returning results immediately.

It worked pretty well, but there was a lot of overhead due to the demuxer requiring frequent back-and-forth calls — after every processing churn, we had to wait for the demuxer to return its updated status to us on the main thread.

This only took a fraction of a millisecond each time, but a bunch of those add up when your per-frame budget is 1/30 (or even 1/60) second!

 

I had been intending a bigger refactor of the code anyway to use separate emscripten modules for the demuxer and audio/video decoders — this means you don’t have to load code you won’t need, like the Opus audio decoder when you’re only playing Vorbis files.

It also means I could change the coupling, keeping the demuxer on the main thread and moving just the audio/video decoders to workers.

This gives me full speed going back-and-forth on the demuxer, while the decoders can switch to a more “streaming” behavior, sending packets down to be decoded and then displaying the frames or queueing the audio whenever it comes back, without having to wait on it for the next processing iteration.

The result is pretty awesome — in particular on older Windows machines, IE 11 has to use the Flash plugin to do audio and I was previously seeing a lot of “stuttery” behavior when the video decode blocked the Flash audio queueing or vice versa… now it’s much smoother.

The main bug left in the worker mode is that my audio/video sync handling code doesn’t properly handle the case where video decoding is consistently too slow — when we were on the main thread, this caused the audio to halt due to the main thread being blocked; now the audio just keeps on going and the video keeps playing as fast as it can and never catches up. :)

However this should be easy to fix, and having it be wrong but NOT FREEZING YOUR BROWSER is an improvement over having sync but FREEZING YOUR BROWSER. :)

MediaWiki audio/video support updates

Mirror of mailing list post on wikitech-l and related lists

I’ve been passing the last few days feverishly working on audio/video stuff, cause it’s been driving me nuts that it’s not quite in working shape.

TL;DR: Major fixes in the works for Android, Safari (iOS and Mac), and IE/Edge (Windows). Need testers and patch reviewers.

ogv.js for Safari/IE/Edge

In recent versions of Safari, Internet Explorer, and Microsoft’s upcoming Edge browser, there’s still no default Ogg or WebM support but JavaScript has gotten fast enough to run an Ogg Theora/Vorbis decoder with CPU to spare for drawing and outputting sound in real time.

The ogv.js decoder/player has been one of my fun projects for some time, and I think I’m finally happy with my TimedMediaHandler/MwEmbedPlayer integration patch for the desktop MediaWiki interface.

I’ll want to update it to work with Video.js later, but I’d love to get this version reviewed and deployed in the meantime.

Please head over to https://ogvjs-testing.wmflabs.org/ in Safari 6.1+ or IE 10+ (or ‘Project Spartan’ on Windows 10 preview) and try it out! Particularly interested in cases where it doesn’t work or messes up.

Non-JavaScript fallback for iOS

I’ve found that Safari on iOS supports QuickTime movies with Motion-JPEG video and mu-law PCM audio. JPEG and PCM are, as it happens, old and not so much patented. \o/

As such this should work as a fallback for basic audio and video on older iPhones and iPads that can’t run ogv.js well, or in web views in apps that use Apple’s older web embedding APIs where JavaScript is slow (for example, Chrome for iOS).

However these get really bad compression ratios, so to keep bandwidth down similar to the 360p Ogg and WebM versions I had to reduce quality and resolution significantly. Hold an iPhone at arm’s length and it’s maybe ok, but zoom full-screen on your iPad and you’ll hate the giant blurry pixels!

This should also provide a working basic audio/video experience in our Wikipedia iOS app, until such time as we integrate Ogg or WebM decoding natively into the app.

Note that it seems tricky to bulk-run new transcodes on old files with TimedMediaHandler. I assume there’s a convenient way to do it that I just haven’t found in the extension maint scripts…

In progress: mobile video fixes

Audio has worked on Android for a while — the .ogg files show up in native <audio> elements and Just Work.

But video has been often broken, with TimedMediaHandler’s “popup transforms” reducing most video embeds into a thumbnail and a link to the original file — which might play if WebM (not if Ogg Theora) but it might also be a 1080p original which you don’t want to pull down on 3G! And neither audio nor video has worked on iOS.

This patch adds a simple mobile target for TMH, which fixes the popup transforms to look better and actually work by loading up an embedded-size player with the appropriately playable transcodes (WebM, Ogg, and the MJPEG last-ditch fallback).

ogv.js is used if available and necessary, for instance in iOS Safari when the CPU is fast enough. (Known to work only on 64-bit models.)

Future: codec.js and WebM and OGVKit

For the future, I’m also working on extending ogv.js to support WebM for better quality (especially in high-motion scenes) — once that stabilizes I’ll rename the combined package codec.js. Performance of WebM is not yet good enough to deploy, and some features like seeking are still missing, but breaking out the codec modules means I can develop the codecs in parallel and keep the high-level player logic in common.

Browser infrastructure improvements like SIMD, threading, and more GPU access should continue to make WebM decoding faster in the future as well.

I’d also like to finish up my OGVKit package for iOS, so we can embed a basic audio/video player at full quality into the Wikipedia iOS app. This needs some more cleanup work still.

Phew! Ok that’s about it.

ogv.js MediaWiki integration updates

Over the last few weekends I’ve continued to poke at ogv.js, both the core library and the experimental MediaWiki integration. It’s getting pretty close to merge-ready!

Recent improvements to ogv.js player (gerrit changeset):

  • Audio no longer super-choppy in background tabs
  • ‘ended’ is no longer unreasonably delayed
  • various code cleanup
  • ogvjs-version.js with build timestamp available for use as a cache-buster helper

Fixes to the MediaWiki TimedMediaHandler desktop player integration (gerrit changeset):

  • Post-playback behavior is now the same as when using native playback
  • Various code cleanup

Fixes to the MediaWiki MobileFrontend mobile player integration (gerrit changeset):

  • Autoplay now working with native playback in Chrome and Firefox
  • Updated to work with current MobileFrontend (internal API changes)
  • Mobile media overlay now directly inherits from the MobileFrontend photo overlay class instead of duplicating it
  • Slow-CPU check is now applied on mobile player — this gets ogv.js video at 160p working on an old iPhone 4S running iOS 7! Fast A7-based iPhones/iPads still get 360p.

While we’re at it, Microsoft is opening up a public ‘suggestion box’ for Internet Explorer — folks might want to put in their votes for native Ogg Vorbis/Theora and WebM playback.

LG G Watch first look

At the Google I/O conference this week they handed out Android Wear watches to attendees; I got the LG G Watch and have been gleefully wearing it for about a day.

The good:

  • There’s no annoying branding on the watch face, and no side buttons to get caught on things.
  • Actual pairing is pretty straightforward using the Android Wear app (once you get it installed…)
  • Showing notifications from my paired Android phone “just works”: texts, Facebook replies, “time for next meeting” pings, etc. It’s also easy to configure it to disable buzzing/pinging on the phone when the watch is active.
  • If you are brave enough to turn on Gmail notifications, you can easily archive a mail from your watch and never have to read it! Or you can swipe away the notification and read it later, if it’s like IMPORTANT or something.
  • Gestures for control are relatively simple.
  • The usual suspects in voice recognition like setting reminders and alarms work, as with Google Now or Glass.
  • If a voice command doesn’t match anything, it does a Google search. Some specific kinds of queries will give results from their knowledge graph, but you can easily end up with generic search results … which inevitably include Wikipedia. :D If you want to actually read a page, it doesn’t try to force it onto the tiny watch screen — it opens up the browser on your phone.
  • The always-on dim display mode looks pretty good indoors or in the shade.

The bad:

  • The screen is nearly illegible in direct sunlight, even pumped up to the max brightness.
  • The G Watch does feel a bit clunky — it’s just kinda big for a watch. But really, it doesn’t feel any worse than my Casio calculator watch did when I was 12. ;)
  • Had to read the directions to see how to turn it on (attach it to the charging cradle and it turns on automatically).
  • Setup currently requires opting in to some prerelease versions of a few Android packages. This presumably will be improved shortly!
  • Voice recognition is a bit spotty. I do wonder what the NSA thinks of my reminder to “resell my medications” (that was “refill”, silly Google!)
  • I wish there were more options for Gmail notifications, namely “mute” and “report spam”.

The ugly:

  • The 280×280 screen resolution looks rather blocky compared to today’s high-end circa-5″ 1080p phones. This is probably a tradeoff for battery life — the watch is already thicker than I’d like, and I wouldn’t want them to have to make the battery huger to last through a day!
  • I had some trouble with the device losing connectivity a couple of times; resetting BlueTooth off and on on the phone seemed to resolve it.

Still have to try:

  • Phone answering — I don’t make or receive a lot of voice calls so haven’t actually tried this yet. Not even sure if it just tells me to grab my phone or if it does some magic watch-speakerphone thing. Who knows? Time will tell.
  • App development — there seem to be two ways to go; either enhanced notifications, or native apps that run on the watch (like the little compass app or the pedometer). Haven’t tried yet, but downloaded the SDK…

Wikipedia new Android, iOS apps in development: current demo

Latest demo screencast of Android and iOS new Wikipedia mobile apps in development — now with login and basic editing! I think folks are really going to like what we’ve done once we get these polished up and out in the stores replacing our old mobile app.

If the on-wiki embedded video player doesn’t work for you, try a mirror on YouTube.

Thanks to Android 4.4’s built-in screen recording feature this didn’t require any custom hardware kit to record, but I had to jump through some hoops to edit it in PiTiVi… I’ll write up some notes on-wiki.

Samsung Galaxy S4 Mini Duos first look (GT-i9192)

So last week was Wikimania 2013 in Hong Kong — it was as always good smashing fun, but it was also one of the first trips I’ve been on where I used a local data SIM in my primary smartphone.

This turned out to be really useful for communicating via online tools (email and Facebook mostly, though some of the other folks used Google Hangouts for group planning), and also let me post some photos during the trip — still have to go over them and do a final push to Flickr! (Previously I’d used an old spare phone for data but it wasn’t able to take decent photos etc.) But on the downside, with the local SIM in nobody could reach me at my regular phone number…

This got me thinking about dual-SIM card phones that can be online on two mobile networks at once — such as to use local data while roaming on your regular home number. Though I travel fairly regularly for conferences, I’d never gotten one before as I’d only seen Chinese brands that I don’t know and am not quite sure I’m willing to trust for software updates. :) At our last hackathon in Amsterdam I carried two phones, one for calls and one for data… effective but bulky.

Turns out there’s a dual-SIM version of the Samsung Galaxy S4 Mini, which I was able to easily obtain in the US via Amazon.

First impressions: size matters

The S4 Mini is definitely smaller than is the trend for the more top-of-the-line phones these days… IMO this is a good thing, as I’ve been whinging about the Nexus 4 being slightly too large to comfortably use in one hand, and the “regular” versions of the S4 and HTC’s One are even huger. Instead, it’s about the size of the older Nexus S; a little wider than an iPhone 5 but about the same length.

This is waaaay more comfortable to use in one hand, though the S4 Mini is also very lightweight — so much so that I’m having to get used to not flipping it over by mistake by being too rough with it. :)

I may actually like using this as a primary phone for the compactness; certainly it’ll be a win for travel — especially compared to carrying two phones!

Display: meh

AMOLED 960×540 screen. Not super great, but functional and sharp enough to read comfortably… the screen is one of the biggest complaints I’ve seen in other reviews of the S4 mini family, though. Personally I’d have preferred a 720p LCD panel if I could make an ‘ideal’ phone in this size to my specifications.

User interface: why, Samsung, why?

Most of the Android devices I’ve personally owned have been Nexus models, running a stock Google user interface, which is to say, a really nice interface since the 4.0 release. This …. is not one of those phones.

The S4 Mini sports Android 4.2.2 Jelly Bean with Samsung’s customized “TouchWiz” interface modifications. It also has hardware home/menu/back buttons which means slightly different gestures versus the software buttons on the Nexus 4.

(My personal pet peeve with the hardware buttons: Android 4.x doesn’t show an overflow menu icon on apps’ action bars. You’re back to the Android 1/2 paradigm of “there might be a menu, try the button and find out!” Bleh.)

Luckily many of the custom behaviors can be disabled, and the standard Google on-screen keyboard can be installed to replace Samsung’s slightly-different version. This helps immensely in making me comfortable using the device.

The dual SIM feature

I was of course most interested in that dual-SIM feature. To test, I put my regular T-Mobile SIM in slot 1, and my Hong Kong temporary data plan’s SIM in slot 2. There’s a ‘SIM Manager’ app which lets you easily select whether you want one, the other, or both SIMs active, and switch the primary data and voice networks between them. The HK SIM came up on roaming, and I was able to easily switch it off until next time it becomes relevant. :)

Unfortunately this feature may also be the thing preventing me from a stock Android experience — there currently isn’t a CyanogenMod build available for this particular model, and I don’t know whether dual-SIM features are available in CM or other more ‘stock’ alternate ROMs. :(

Storage

8 GB of total internal storage (about 5 GB available after the OS), plus a micro-SD card slot (welcome back to manual storage management!) Adequate core for a casual phone and extensible for media storage, hey that ain’t bad.

Optional encryption is supported, enabled separately for the core storage and the SD card.

CPU/GPU/RAM/Blah

Fast enough for my basic needs. See benchmarks in generic S4 mini reviews on the interweb if you like that sort of thing. :)

Next steps

I’ll poke at this phone for a couple weeks and see how I like it; may or may not switch back to the Nexus 4 as a primary phone, but definitely this is going to be coming with me on international trips!

 

 

Ubuntu Touch first look

An early developer preview of Ubuntu Touch for Nexus phones and tablets has been released! Since I’m still waiting on the Nexus 4 I’m reserving for Ubuntu testing, I temporarily installed a tablet image on my Nexus 10.

device-2013-02-22-144057

Be warned: this is a very early preview and is more demo-ware than usable operating system at this point. Don’t flash it on your primary phone/tablet or you’ll be mighty disappointed, I suspect. :)

The good:

  • It looks very pretty. They’ve got some good UI designers. :)
  • Edge swipe gestures leave more of the screen available for applications than Android does, as with some other new OSs like Windows 8/RT and BlackBerry 10.
  • Seems to have a WebKit-based browser as default. Renders Wikipedia nicely — we get the desktop version by default on the Nexus 10 as it masquerades as an iPad, which we also send to the desktop site.
  • Some of the hardware actually works, like wifi networking and the cameras. :)
  • There seems to be infrastructure for inter-application data sharing.
  • It handles high-density screens from the get-go; the 2560×1600 Nexus 10 screen looks awesome. Text and photos are crisp and beautiful.

The bad:

  • Very unstable. The UI has crashed on me several times in just a couple of hours of testing. This kicks you back to the login screen.
  • No screen rotation on orientation change.
  • On-screen keyboard is pretty flaky; while it comes up and goes away much more reliably than the GNOME 3 or Ubuntu desktop versions, it fairly often fails to register keypresses.
  • Scrolling and similar UI operations are sometimes a bit sluggish, especially in the browser. This may be due to insufficient graphics acceleration at this stage.
  • WebM and Ogg video don’t appear to fully work in the browser… at least I can’t get anything to play on Wikimedia Commons.

The ugly:

  • Outside the browser and anything you install yourself with Qt Creator, there aren’t a lot of useful apps at this stage. There’s an “Available for download” section on the apps home screen, but tapping the icons does nothing. Aggravating as one of them is ‘Wikipedia’ and I want to know what it does. :)
  • Not clear how traditional Linux CLI and GUI programs will work. Will they run on the tablet-style “desktop” or will it fire up a classic desktop-style UI when you dock? There’s no terminal app yet; you can get a shell via the Android debug bridge tools and can set up SSH from there, but that doesn’t get you into the GUI.
  • No mobile data support for the phone builds yet; data is wifi only.
  • The current images include non-free drivers and sample data, so isn’t fully open source yet. Ick!

I haven’t tested HDMI output yet.

Development

I haven’t done more yet than firing up Qt Creator and fiddling with the demos on the desktop, but it’s interesting to see the development landscape shaping up.

Ubuntu Phone/Touch’s “native” UI toolkit is Qt, with emphasis on using ‘Qt Quick’ and QML+JavaScript for rapid development. It also allows for C++ Qt apps, including low-level OpenGLish things. There’s a sample port of some game that… doesn’t seem to work yet. ;)

Qt is a bit surprising as Ubuntu’s traditionally been GNOME/GTK+-focused. But it makes sense; it’s capable, developer-friendly, now has the declarative stuff like QML markup and first-class JavaScript support, and most importantly it’s also in use on other platforms like BlackBerry 10 and the (mostly dead or is it?) Meego. This could help with code sharing on ‘alternate platforms’… OpenGL-ish games probably will see this as no worse than Android, or possibly easier since there’s no Java/JNI stuff to play with for native apps.

Compared to Android

At this stage there’s no comparison to Android; you just can’t get much done. But we’ll see as it matures…

One of the selling points is that Ubuntu Touch is supposed to be dockable and usable as a “real computer” when connected to a mouse and keyboard and external screen. However this doesn’t seem to be in the current versions, and it’s unclear yet how much better it will be than Android when docked. (Android also supports keyboards, mice, and external screens… but doesn’t as conveniently run existing Linux apps.)

Compared to Firefox OS

The ability to write “native code” apps is probably going to be a big mark in favor of Ubuntu for things like games, which tend to be C/C++ and OpenGL-heavy. Though Mozilla’s done a lot of work with things like emscripten and WebGL making it possible to port C/C++ game engines to JavaScript, I suspect performance will always lag a bit on Firefox OS.

Ubuntu Touch also definitely supports high-DPI displays and tablet-size screens at this stage; it’s unclear to me whether Firefox OS is ready for high-DPI displays or larger screens. Even if it runs all the images in the apps are low-resolution so a lot of stuff might look blurry.

However, Firefox OS is much further along. It’s got a working app marketplace, a browser that doesn’t crash quite as constantly, and most importantly… has hardware partners who are already committed to shipping phones this year.

It looks like this version of Ubuntu Touch was kept as a skunkworks project for a while, then rushed out so it could be demoed at Mobile World Congress. Hopefully development will be more open from here out and it will improve rapidly.