Month: March 2016

Web Modules for JS + asset files: why don’t you exist yet?

Ugh, there really needs to be a sane way to package a “web” module that includes both JavaScript code and other assets (images, fonts, styles, Flash shims, and other horrors).

I’m cleaning up some code from my ogv.js JavaScript audio/video player, including breaking out the audio output and streaming URL input as separate modules for reuse… The audio output class is mostly a bit of JavaScript, but includes a Flash shim for IE 10/11 which requires bundling a .swf file and making it web-accessible.

Browserify can package up a virtual filesystem into your JS code — ok for loading a WebGL shader maybe — but can’t expose it to the HTML side of things where you want to reference a file by URL, especially if you don’t want it preloaded.

Bower will fetch everything from the modules you specify, but doesn’t distinguish clearly between files that should be included in the output (or what paths will be used to reference them) and random stuff in your repo. They kinda recommend using further tools to distill down an actual set of output…

Dear LazyWeb: Is there anything else in the state of the art that can make it a little easier to bundle together this sort of module into larger library or app distributions?

Windows RT is dead, long live Windows 10

Some readers with niche interests may remember Windows RT, the unpopular now-discontinued flavor of Windows 8/8.1 for ARM-based tablets.

Windows RT shipped on the first two generations of Microsoft Surface tablets (the lower-end Surface RT and Surface 2, not the Intel-powered Surface Pro line), and a tiny handful of third-party ARM tablets.

Even though it exclusively ran on touchscreen tablets and couldn’t run any of the massive library of 3rd-party Win32 software, Windows RT had the same widely-hated hybrid touch/desktop interface that laptop and desktop users encountered on Windows 8. Microsoft’s inability to port its Office suite fully to the touchscreen environment in time for Windows 8/RT’s release particularly highlighted this divide, as you had to bounce into the desktop every time you pulled up Word or PowerPoint.

It was pretty frustrating to use, and pretty frustrating to develop for — since the desktop was included, there was no technical reason why Win32 application developers couldn’t recompile their apps for ARM processors (maybe with some touch-friendly UI enhancements) like Microsoft did with Office… but Windows RT, unlike Windows 8, was locked down to require code signing approval from Microsoft on both touch-style apps and Win32 apps.

So developers had to port their apps completely to the new sandboxed ‘Modern’ API subsets and package format in order to run on Windows RT at all… and on Windows 8/8.1 desktops, those apps would only run fullscreen, not in… you know… windows. Overwhelmingly developers chose not to jump through those hoops, and everybody lost.

When it was announced that there would be no upgrade path for the ARM-based Windows RT devices to Windows 10, that was the final nail in the coffin.

Or was it?

The true innovation of Windows RT is that it was Windows 8’s ARM flavor. The API surfaces were the same, the source code was the same, the security updates came out together, and the major 8.1 upgrade came out together. If you were one of the few who developed ‘Modern’ apps for Windows 8/8.1, you only had to not uncheck the ‘ARM’ box when building and deploying to the Windows Store to reach Windows RT users as well.

That it was exposed to end-users in the Windows 8 ‘weird hybrid interface’ stage, and without Win32 access for 3rd-party developers, is perhaps unfortunate. But then we can say that of Windows 8 as a whole. ;)

But what this experiment enabled was the replacement of the old Windows CE + weird .NET-based Windows Phone 7 with the Windows NT + weird .NET-based Windows Phone 8, followed finally by the Windows NT-native Windows 10 Mobile and Windows 10 IoT… and even the Xbox One, which is picking up more user-facing Windows 10 features with each major update.

Like Windows RT  & Windows 8, Windows 10’s Mobile/IoT editions are built from the same base as Windows 10 for Intel-based desktops/laptops/servers, just in ARM flavor. Application developers targeting the Windows 10 ‘Universal Windows Platform‘ can build and deploy the exact same binary package on all flavors, with compilation targeting the CPU architectures rather than the device types.

And unlike Windows RT & Windows 8, the various Windows 10 flavors actually tailor their user interfaces to their environments. :)

Windows 10 has also reduced the lock-in requirements a bit, making it possible for any end-user to relax the code-signing requirements for Universal Windows apps to install custom apps.

Now, it may be that Microsoft has dropped the ball and will never recover in the phone market. But having a common platform base between desktop/laptop, tablet, phone, TV/game, and ‘other device’ seems like it will be an advantage in enabling cross-development now that a subset of Surface-branded tablets isn’t the only reason to touch the ‘Modern’ APIs.

The real question is going to be how hard it is to get into the Universal Windows Platform ecosystem at all — once you’re in, handling multiple devices gets easier. Rumored tools to make it easier to port Win32 apps to run within UWP should help with this, as may things like the Objective-C bridge for devs with iOS codebases.

Google seems to be doing much the same with Android, trying to expand its Linux + weird Java system from phones & tablets to smartwatches and TVs/game consoles. It remains to be seen whether Android will also push out or merge with Chrome OS in Google’s laptop/desktop market.

In comparison, Apple is pursuing a slightly different strategy where various Darwin flavors (Mac OS X, iOS, tvOS, watchOS) are closely related but slightly different, requiring separate compilation and deployment for every flavor. You can usually build from the same source using the same tools, but it feels a lot more fiddly to me.

 

 

Windows 10 Objective-C bridge

While I was waiting for updates to download I finally took a look at Microsoft’s Objective-C ‘bridge’ for porting iOS apps to the Windows 10 platform.

It’s technically a pretty nice system; they’re using clang to build your Objective-C code natively to the Universal Windows Platform as native x86 or ARM code, with an open-source implementation of (large parts of) Apple’s Foundation and UIKit APIs on top of UWP. Since you’re building native code, you also have full access to the Windows 10 APIs if you want to throw in some ‪#‎ifdefs‬.

I suspect there are a lot fewer ‘moving parts’ in it than were in the ‘Project Astoria’ Android bridge (which would have had to deal with Java etc), so in retrospect it’s not super surprising that they kept this one while canceling the Android bridge.

I don’t know if it’ll get used much other than for games that targeted iOS first and want to port to Xbox One and Windows tablets easily, but it’s a neat idea.

Probably tricky to get something like the Wikipedia app working since that app does lots of WebView magic etc that’s probably a bad API fit… but might be fun to play with!

Wikipedia data dumps future thoughts

There’s some talk & beginnings of work planning on a major overhaul of the Wikipedia/Wikimedia data dumps process.

The basic data model for the main content dumps hasn’t changed much in 10 years or so, when I switched us from raw blobs of SQL ‘INSERT’ statements to an XML data stream in order to abstract away upcoming storage schema changes… fields have been added over the years, and there have been some changes in how the dumps are generated to partially parallelize the process but the core giant-XML-stream model is scaling worse and worse as our data sets continue to grow.

One possibility is to switch away from the idea of producing a single data snapshot in a single or small set of downloadable files… perhaps to a model more like a software version control system, such as git.

Specific properties that I think will help:

  • the master data set can change often, in small increments (so can fetch updates frequently)
  • updates along a branch are ordered, making updates easier to reason about
  • local data set can be incrementally updated from the master data set, no matter how long between updates (so no need to re-download entire .xml.bz2 every month)
  • network protocol for updates, and access to versioned storage within the data set can be abstracted behind a common tool or library (so you don’t have to write Yet Another Hack to seek within compressed stream or Yet Another Bash Script to wget the latest files)

Some major open questions:

  • Does this model sound useful for people actually using Wikipedia data dumps in various circumstances today?
  • What help might people need in preparing their existing tools for a switch to this kind of model?
  • Does it make sense to actually use an existing VCS such as git itself? Or are there good reasons to make something bespoke that’s better-optimized for the use case or easier to embed in more complex cross-platform tools?
  • When dealing with data objects removed from the wiki databases for copyright/privacy/legal issues, does this have implications for the data model and network protocol?
    • git’s crypto-hash-based versioning tree may be tricky here
    • Do we need a way to both handle the “fast-forward” updates of local to master and to be able to revert back locally (eg to compare current and old revisions)
  • Technical issues in updating the master VCS from the live wikis?
    • Push updates immediately from MediaWiki hook points, or use an indirect notify+pull?
    • Does RCStream expose enough events and data already for the latter, or something else needed to ‘push’?
    • Can update jobs for individual revisions be efficient enough or do we need more batching?

Things to think about!