Category: android

Screencasts of mobile devices with HDMI capture

When trying to post about behavior of web or mobile applications, a picture is worth a thousand words… but a video can be worth a thousand pictures. Videos can capture complex behaviors that are hard to fit into a screenshot, can be narrated, and show actual performance as well as the look of something.

Here’s a screencast of the upcoming Wikimedia Commons app importing a PDF from Keynote on an iPod Touch:

I’d stumbled upon some references to a performance monitoring tool called Eideticker that Mozilla’s using for the Android version of Firefox. This uses an HDMI video capture card to record live video output from a Galaxy Nexus or other phone, and can then mark timing of various on-screen events down to 1/60 second precision.

‘Blackmagic’, the company that makes the PCIe card Mozilla uses, also makes Thunderbolt-connected standalone widgets that are easier to connect to a MacBook Pro. I picked up the “Intensity Extreme“, which hides the (for me) unneeded analog video ports behind a breakout cable that I can stuff in a drawer.

See details on which devices I’ve gotten to work and more sample casts.

Limitations:

  • Capture works in their utility program and various high-end editing programs via plugin, but it doesn’t appear as a camera to other apps. So it’s not ideal for live screencasting on a Google Hangout or such, but could probably be pressed into service.
  • Captured video is uncompressed, so files are large!
  • No 1080p60 capture — they have a higher-end model that can do this, but I didn’t shell out the dough. Most devices I’ve tested output at 720p60 or 1080p30 though, so that’s not a problem.
  • No autodetection of the resolution — you have to pick the right resolution in the capture utility, possibly by trial-and-error.
  • No passthrough for the Thunderbolt connection.
  • Must purchase HDMI and Thunderbolt cables separately. :)

Responsive images for high-DPI displays in MediaWiki

I’ve been working on support for natively showing suitable images for high-DPI displays in MediaWiki, starting with basic content images.

On desktops and tablets there’s just a few devices like the Retina MacBook Pro that need this, but in the mobile world there are loads of devices at 1.5x or 2.0x the traditional resolutions.

You can see a live demo of this patch in action on these test articles:

Here are some screenshots from an iPhone 5 simulator showing fragments of the San Francisco Wikipedia article as currently displayed on en.m.wikipedia.org and with my responsive images patch…

These maps are absurdly sharper on the right side:

And this photograph is visibly much sharper on the right, showing detail of the building’s texture and patterns that wasn’t previously visible without clicking through to the detail page:

The current version of the patch uses the ‘srcset’ attribute as defined in the current HTML 5 working group version, only using the density options and not the width/height options. Since browsers don’t yet support this, I also include some JavaScript to check the display density and load the appropriate image from the srcset.

Patch in gerrit:

The current version has been tested with a number of devices and browsers, including:

  • iPhone 3Gs, Safari (low-res 1x)
  • Nexus One, Android browser (medium-res 1.5x)
  • iPod 5th-generation, Safari (high-res 2.0x)
  • Galaxy Nexus, default browser (high-res 2.0x)
  • Galaxy Nexus, Chrome (high-res 2x)
  • Galaxy Nexus, Opera Mobile (high-res 2x)
  • BlackBerry 10 dev alpha, default browser (high-res 2x)

And on a Retina MacBook Pro:

  • Mac OS X, Safari (2x)
  • Mac OS X, Chrome (2x)
  • Windows 8, IE desktop (2x, with desktop zooms set to 200%)
  • Windows 8, IE Metro (2x)

and on an 11″ 1366×768 Windows 8 tablet:

  • Windows 8, IE Metro (1.5x)

Currently unsupported:

  • Opera Mini (we don’t serve any JS)
  • Firefox (can’t yet detect resolution)
  • Windows Phone IE (can’t yet detect resolution)

 

Nexus 7 quick review

First impressions using the Nexus 7 tablet for a couple days…

image

I used to think the 7″ tablet space was a solution searching for a problem. Prior experience with the Kindle Fire and the Blackberry Playbook — both boxy, awkward slates — hadn’t impressed me. The Nexus 7 is much nicer and feels like it was actually designed to be used by humans… the edges are rounded and beveled so it’s actually comfortable to hold.

Hardware-wise things seem nice enough though I’m surprised by the lack of a rear-facing camera. I’ve actually seen a lot of iPads being used as cameras at museums and such, and the smaller tablet would be much… less unwieldy. Front-facing camera exists for I presume video chat, but you can’t really aim with it at anything else.

The screen seems nice; it packs the pixels in more densely than the older 7″ tablets and — thankfully — uses real square pixels instead of the awful hex pixel layout so many of the Nexus series phones have been cursed with. It’s still not as sharp as the iPad 3, but it’s a noticeable bump. I haven’t investigated in too much detail, but the Nexus 7 appears to be using Android’s existing support for ‘hdpi’ screens with a 1.5 multiplier between virtual and physical pixels, so apps already know how to handle it and should include suitable artwork. Downside: rounding oddities sometimes lead to series of lines like borders jumping between 1 and 2 pixels. Yeah, I’m picky!

Like the Kindle Fire, the Nexus 7 really pushes its master’s online store. Like the Kindle Fire, I have little interest in buying stuff from their store directly though, so it remains to be seen whether that will be a big selling point for other people.

(I tend to buy books through the Kindle store since I can read them on multiple devices from multiple manufacturers — the “least invasive” DRM with cross-platform support wins. My music comes sometimes from iTunes, sometimes from Amazon, and sometimes ripped from CDs — yes they still exist! I don’t tend to buy movies or TV in any format, generally preferring rental options or … shudder… cable.)

Amazon app store works – as usual you have to enable installation of apps from untrusted sources… the kind of switch I seriously wish Apple would just put on iOS so people can stop jumping through hoops jailbreaking. I’ve only got a couple games from there such as Plants vs Zombies, which runs noticeably snappier on the Nexus 7 vs the Kindle Fire.

Typing with thumbs in landscape portrait mode is comfortable, and a bit less surprise-prone for me than the Fire where I often hit the close keyboard button by mistake. :)

Encryption option for storage is pleasing, seems to work reasonably transparently as on the Galaxy Nexus.

Android 4.1 seems overall similar to 4.0 with a fancier notification display. Most visible change is Chrome, which replaced the older default Android Browser app. It feels snappy, looks good, and has all the expected WebKit mobile and CSS goodies.

Some of us want to diversify and run a non-WebKit browser. Opera Mobile installs and appears to run OK though I haven’t used it much.

Surprisingly you cannot install the current release version of Firefox for Android on the Nexus 7. You can install beta or nightlies and they work, but it’s showing the phone interface, none of the tablet goodies like tab thumbnails along the left. Disappointing, but should be remedied in time.

Note that just as in Android 4.0, you can take screenshots by holding down the power and volume-down buttons. This is slightly more awkward to do than on the Galaxy Nexus as the buttons are all on the same side. Blah!

So in short: so far I like it, by far the best 7″ tablet I’ve used. Not 100% sure a 7″ tablet is a necessity, but it’s more portable than the 10″ers, including being easier to type on when lying in bed or on the couch… so it comes down to “do you need a tablet anyway?” It seems a nice web & reader machine and I expect to take it on a lot of flights… might actually replace my iPod Touch if I can get some music loaded on. :)

Mostly mobile monkeying

Mostly poked at mobile today – switched it from an iframe to loading into a div, fixing some of our scroll & click through issues and fixing it for Android 4.

Experimented with iscroll 4 library to handle the scrolling and add zoom, but too slow for now.

A little more poking at EmbedScript experiment; I have a sandbox domain temporarily on a Wikimedia Labs VM. Will show off more on this later.

Galaxy Nexus first weekend impressions

My Galaxy Nexus arrived in the mail on Friday; I’ve been fiddling with it intermittently since. (You’ll go blind!)

Some first & second impressions:

  • It’s huge. It feels like being a little kid trying to use some giant remote control… it’s just big enough to feel awkward in my hand, and to make one-handed usage difficult with some thumb stretching. We’ll see if I get used to it…
  • The screen feels slightly smaller than it “really” is thanks to the use of the bottom edge for software buttons replacing the classic Android hardware buttons — so it’s not *as* big as you’d think from ‘4.65″ screen’. But that space is now reclaimable, and it turns out to be useful when watching full-screen video.
  • The extra high-density screen is beautiful! At regular viewing distance, text is almost as sharp as on my iPod Touch’s Retina display. The hex tile layout of the OLED pixels is much less visible than on my older Nexus One.
  • Having decent internal storage and ditching the awful SD card has made storage management much easier — my Nexus One was constantly whinging that it was out of internal storage space despite having gigs free on the SD card.
  • Interactive performance is MUCH snappier than previous devices I’ve used, and comes closer to what iOS usually manages.
  • Sometimes the Google folks are going overboard on icons; it took me a while to figure out what the hell all the buttons in their Gmail app did since they switched them all from nice obvious buttons like “Archive” to mysterious icons.
  • Our Wikipedia Android app is totally broken on Android 4; we knew that on the emulator, but it’s muuuuch easier to test it on the real device. Some things are being fixed upstream in PhoneGap, others we’ll still have to work out…
  • Transferring devices on Android isn’t quite as easy as iOS. With iOS, I can restore a complete backup of all my application data and preferences from one devices to another via iTunes: my new iPod Touch was restored from my old iPhone 3Gs which I can’t even find anymore (it’s in the apartment *somewhere*) and updated pretty smoothly. The Galaxy Nexus was able to restore some settings and applications from the ‘backup to Google’, but not everything. Many apps needed to be manually reinstalled and/or reconfigured for accounts; games have lost high scores and achievements etc. Arranging my apps on my home screen again was a painstaking task, and very annoying.
  • I remember it being a big deal that you can get to the Camera app directly from the lock screen in Android 4 — but apparently this only goes for if you don’t have a …. lock … on the lock screen. Add pattern lock, and that feature’s gone. Camera is not in fact locked down so that you can activate it and take photos without going through your unlock code. D’oh!
  • I CAN TAKE SCREENSHOTS! FINALLY! THIS IS GOOD!!!!!!! Press power and volume-down and hold for a moment; it’ll do a cute visual transition showing your screenshot and give you a notification that lets you call it back up to view or share it.
  • Did I mention it’s really huge? It feels like it’ll fall out of my pocket at any moment. :P
  • Haven’t had much chance to test 3G/4G reception yet, though the rumor is Verizon will do better in the subway than AT&T. We’ll see on Monday!

What I kinda want is the same phone, but with a 4″ diagonal screen at 1024×640. Make it fit the hand better, rather than going for that hugeness. I don’t watch a lot of movies on my phones, but I do hold it in one hand and read/write news & messages every day.

BugTender updates

Can now post comments on bugs! Auth prompt at post time. No offline queuing yet.

Bug list defaults to showing recently filed bugs. Various search and sort options, incomplete but a start. Doesn’t handle huge return lists well; server gives chronic order oldest first, we want the opposite.

Initial appcache for offline usage. Limited as there’s no persistent data cache yet, but you can load the page when offline.

Restructured bug view to put comments first.

More details in bug list items.

20111205-092116.jpg

High-density displays: mobile and beyond

High-density displays are already here on small screens, and should start hitting tablets, laptops, and desktops in the next couple years. Apple’s iPhone 4 and current-model iPod Touch sport a best-of-breed 326dpi display, while lots of Android and Windows Phone devices have an intermediate 240dpi resolution. With the new Galaxy Nexus, Android’s entering the 320+dpi world as well.

The most immediate effect is of course that text renders as sharply on screen as in print. As a web-based application, Wikipedia gets this über-shiny text rendering  “for free”… but for graphics, we’ll have to work a little harder.

Most images you’ll find on the wikis are either largeish raster images that are being scaled down for in-article viewing, or scalable SVG drawings that are being rasterized to PNG on the servers for broadest browser compatibility.

Unfortunately neither automatically takes advantage of a high-density display!

For SVG drawings, we can simply start serving the original SVGs instead of the rasterized PNG images, and they’ll look lovely (images from Gas giant article mobile view, rendered on an iPod Touch, with high-res versions swapped in by this bookmarklet):

We’ll need to make sure we can do this with a good fallback to PNG however — while all major desktop and mobile browsers in their *latest* versions support SVG, there’s still a lot of Internet Explorer users and Android users who can’t view SVG. Some less capable mobile devices may also be unable to handle full SVG (Android is rare among smartphones; SVG support is missing on most Android phones today but is present in Android 3.x tablets and upcoming 4.x phones).

We also need to make sure that file sizes and rendering times are reasonable; large maps might be very expensive to transfer and render client-side when we really only need a small map. See my SVG-Open 2009 slides for some data on PNG vs SVG sizes and the need for work on shrinking files.

 

Other files are usually available in a size larger than they’re shown on default-density screens, so putting a 1.5-sized or double-sized thumbnail into the same screen space makes it match the screen’s actual density. This is particularly useful for diagrams that were created as raster images instead of SVGs, since they often contain fine lines and text which benefits from the full resolution:

SVG and PNG images used for icons and diagrams are probably the highest-priority for high-density images. Photographic images can also benefit, but usually scale up better than line drawings and text because they have fewer sharp edges to begin with.

For raster images of all types, doing dynamic replacement is extra work; CSS or JavaScript can be used in various circumstances but it can take work to make existing code using <img> tags “just work” without loading up extra versions of images, or loading higher resolutions than you need on other devices.

Math equations are another thing; currently we render most math equations to PNG images (via LaTeX); these also render poorly compared to other text on a high-density display. The best solution for this appears to be to migrate towards client-side math rendering based on the MathJax library; using MathML or HTML and CSS, this system is able to use the browser’s native text rendering much more effectively for math.  Equations rendered this way look stunning at high resolution!

 

Introducing BugTender: mobile Bugzilla frontend

I’ve been looking for a good solution to catching up on bug reports on my morning commute. The regular Bugzilla user interface doesn’t scale down well to mobile, and using the regular web interface is too flaky when connectivity goes in and out.

Enter BugTender: a preliminary HTML web app with a mobile-friendly interface to browse (and later, comment on & triage) your bugs.

Using HTML and jQuery Mobile lets the app run on multiple platforms and browsers — it won’t be tied to just WebKit, and works in a regular desktop browser as well. Wrapping that in PhoneGap for a ‘native app’ will allow further integration, such as access to the camera and photo gallery on iOS and older versions of Android — think uploading screenshots straight from your phone about problems in the browser!

Anyway, that’s how I spent chunks of my Thanksgiving weekend… more to come. :)

 

Amazon Kindle Fire notes

Driven by curiosity, I went ahead and preordered an Amazon Kindle Fire tablet, which I’ve been using off and on for the last week.

Wiki-specific notes:

  • Wikipedia (regular site) is on the default bookmarks, and displays nicely in the Silk browser (though the fundraising banners don’t quite fit in portrait layout!)
  • Our in-progress Android app mostly runs fine, though the small-screen layout feels slightly wrong on this 7″ screen. Will benefit from a more tab-oriented interface that can use sidebars or such. ‘Near me’ geo search doesn’t work, as it relies on Google Maps APIs that aren’t present on this device.

I’ll leave super review details for dedicated reviewers, but here’s a few notes from me:

  • customized UI mostly looks fairly polished, but has a few rough edges
  • performance is middling. reading, web, video playback are acceptable enough. Some audio skipping when playing & downloading music simultaneously (boo!).
  • battery life seems acceptable enough (not thoroughly tested, just using it intermittently)
  • A few parts of the UI are actually very smoothly animated, such as the ‘cover flow’-like bookshelf on the home screen — however it’s not synchronized to vertical refresh, and there’s visible tearing. Points for near-60fps animation, but points off for the tearing.
  • Silk browser seems pretty much like the stock Android browser with tabs. Disabling the server-assisted acceleration is easy in prefs.
  • Kindle book reader looks and behaves pretty much like the stock Android Kindle app but with some customized menus. It also can view comics/graphic novels which don’t appear to be available on Android/iPhone/Cloud Reader yet. :(
  • Built-in store stuff looks & works well; nicely designed to funnel all your money into Amazon. ;) They gave me a “free month” of Amazon Prime subscription, designed to get me used to streaming video, but I’ve already got Netflix, Apple, Comcast, and who knows what else available so who cares.
  • Since the Fire lacks hardware back/home/menu buttons, it has a software toolbar which displays these, similar to the software buttons in Android 3 & 4. Unfortunately it’s poorly laid out, centering most of the buttons so they’re hard to reach with your thumbs while holding the device.
  • Firefox runs, but isn’t yet available in the Amazon app store and the main download links herd you to the Android Market which isn’t there. There are manual download links which work though.
  • the onscreen keyboard is very comfortable in portrait mode when holding the tablet with two hands; I can type with the thumbs, and it’s easier than on a tiny phone. In landscape mode however the keys are too widely stretched to reach the middle comfortably.

On the software freedom front: the Fire actually fares better than I expected. Like most Android devices it defaults to allowing only apps from ‘known sources’ to install (in this case, the Amazon App Store) but this can be flipped easily in preferences, and you can install .apk packages from direct download or USB transfer.

Hooking up to a computer for debugging over USB works but requires some tweaks to get the Android SDK to recognize it. This is the only way to do screenshots, since self-hosted screenshots weren’t added until Android 4.0 (grrrr! my fave little feature from iOS since the 2007 iPhone release).

As a practical matter, lack of access to Google’s Android Market is inconvenient; even many free apps haven’t set themselves up on the Amazon store and don’t offer direct downloads. Additional alternate app installers can be used as well though, such as F-Droid or the various custom markets folks have been using in some areas.