Beta testers for StatusNet Mobile wanted soon…

A few weeks ago, we released the first public beta of StatusNet’s dedicated desktop client for Linux, Mac, and Windows.

We’re still working on bug fixes and improvements, but we’ve also been working on a mobile version, which runs on Android and iPhone/iOS platforms (Blackberry support isn’t ready yet in the Titanium cross-platform runtime we’re using, but it should come along in a few months). If you’re really brave you can dive into the source code, but we’re in the middle of major restructurings and UI design so it’s not really usable yet. :)

Once we get the UI polished up in the next week or so, we are going to need some testers to help make sure things work on different devices, different OS versions, at different screen sizes, with different servers, etc…

If you have an iPhone or iPod Touch and want to test without being a developer yourself, you’ll need to find your device’s UDID number and send it to us; due to the way Apple’s code signing works you won’t be able to install the beta distribution unless your device’s ID was included when building the signing key.

Android users will only need to ensure that “Unknown sources” is checked in Application settings, which allows installing apps from the web as well as from the Android Market. (Some models unfortunately don’t allow changing this setting, in which case you may need to root your phone to get the beta running. Sorry!)

Particular things we’re looking for…

  • older iPhone 2G, 3G devices still running iPhoneOS 3
  • Android phones running older OS versions (1.6 or later; 1.5 does not work with our runtime)
  • Android devices with unusual screen sizes:
    • small low-res screens (less than 320×480)
    • larger tablet devices

Gitorious ain’t glorious today

Gitorious’s ssh-authenticated access has been down all day, preventing us from pushing our commits on StatusNet and the StatusNet client.

Until it’s fixed, I’ve put up my StatusNet & StatusNet Desktop/Mobile client dev branches up on github, to be merged back onto the Gitorious mainline later:

* StatusNet – http://github.com/brion/statusnet
* client – http://github.com/brion/statusnet-client

Hopefully this inconvenience won’t last too long; I can add folks as collaborators on those repositories to make direct commits in the meantime.

Introduction to OStatus: Atom-based federated messaging

I’ve written up the little introduction to StatusNet’s new inter-site federation protocol; for now you can also find this article on the StatusNet wiki. I’ll follow up in the next couple days with demos of the subscription UI and subscribing to Google Buzz feeds as native OStatus data sources. — brion

What is OStatus?

You’re about to learn what OStatus is and how it’s going to make communications between StatusNet sites work better!

Some of you may remember the bad old days when you couldn’t send e-mail between services like AOL and CompuServe. StatusNet/Laconica sites have used a custom protocol OMB to set up subscriptions between sites, but OMB requires special support to be used with non-StatusNet services and is hard to extend.

By building on top of shared technologies already in use and being rolled out, we can go beyond that… our new effort, with better compatibility between sites, is named OStatus.

The basics

If we step back a bit, we find that Atom and RSS feeds are already a lingua franca between sites. Virtually every service provides such feeds, giving a way for other tools and services to read someone’s outgoing posts.

File:Basic-feed.pngAt its simplest, a feed is a list of recent posts with some author attribution.

But the fun of social networks is in real-time interaction — if your updates don’t get sent to your friends right away, it’s just not the same.

Luckily, that exact problem has been solved recently, with PubSubHubbub (PuSH). In the PuSH system, a site can subscribe to updates for a feed from a hub server associated with that feed. Whenever a new post is created, the publishing site pings the hub, and the hub sends out just the new posts to all the subscribers.

OStatus is built on that simple base: each participating site produces Atom feeds of updates and uses PuSH subscriptions to send relevant updates to other sites:

File:Push-delivery.png

Alice posts a message on identi.ca; a fragment of her Atom feed is PuSHed out to the other StatusNet sites that have users following her posts. Each of those sites is responsible for routing the message to individual subscribers.

The real beauty of it is that at this point we already have something useful, without anything StatusNet-specific. In fact you can already subscribe to someone’s public Google Buzz feed as an OStatus remote user, and they haven’t done anything special for us!

The good stuff

PuSH is a low-level piece of the puzzle; it’s just a way for services to send updates to each other. For a fully interactive social network, you need that human touch… you want to know who’s subscribing to you, which of your posts they like, and you want to get their replies to you.

For this we’re making use of more extensions that build on top of Atom feeds: Activity Streams, Portable Contacts, and Salmon.

Activity Streams extends Atom feeds to describe social “activities” more specifically than just “ok here’s a post”. Entries are marked with a verb (post, follow, favorite, etc) and more detail on who or what’s being acted on. We’ve extended this further using elements from Portable Contacts to provide profile information:

Extended author information beyond what Atom’s native <author> element provides.

Salmon was created as a way for blog aggregators to send comments back to the original blog server, “like salmon swimming upstream”. We’re using this channel to send replies as well as other user-to-user events:

File:Salmon-delivery.pngBob replies to Alice from his home site; since she’s not subscribed to him, the reply is sent back to identi.ca as a Salmon ping.

When your friend on Example.Net subscribes to your Identi.ca account, Example.Net sends an Atom entry with an Activity Streams “follow” verb to your Salmon reply channel on Identi.ca. This lets Identi.ca know who subscribed to you, so their profile gets added to your subscribers list and you get a notification that they’re following you.

If your Example.Net buddy gets social network overload and decides to unsubscribe from you, another notification comes through about the unsubscribe, and we can drop them from the list.

Other events can be sent over this channel as well, such as when your friends on other services mark your posts as a favorite.

Groups

One of the most annoying limitations with OMB 0.1 was that there’s no support for joining a group on another service. You might have noticed that groups have Atom feeds of posts just like users do… if you were wondering if that means groups could be run over OStatus too, you’re correct!

As a user on Identi.ca you can join a group on Example.Net without setting up an Example.Net account. Group posts from Example.Net will appear right in your inbox on Identi.ca just like any local group, and you can even post back to it.

Why OStatus will rock your world

Hey everybody!

Things may have looked pretty quiet the last couple weeks, but here at SNI we’ve actually been rushing to get some really awesome things ready for you all…

In addition to finalizing StatusNet 0.9 with scalability and bug fixes to make it release-ready, we’ve been implementing our new federation protocol OStatus, successor to our older OMB 0.1.

I know the scattered notes we’ve got up on the wiki right now are a little confusing, so over the next few days I’m going to start consolidating those into a clearer picture what OStatus is and how it’ll let StatusNet sites connect better to each other and to other systems.

In the meantime, active work on OStatus is currently proceeding in the ‘testing’ branch in our gitorious mainline repository…

Stay tuned!

— brion vibber (brion @ status.net)
Software Architect
StatusNet, Inc
San Francisco

XMPP output fix for StatusNet

Yay, it’s big commit time!

I’ve landed my XMPP output queuing work in 0.9.x, running on my public test site.  Do let me know if XMPP subscription or i/o is flaky on it!

Should be ready to merge to testing & master and deploy when we’re content to do so… there is a database change necessary for the DB-based queueing system, so I want to confirm that’s not a problem before pushing out.

Big thanks to Craig Andrews for his work on generalizing the DB queues which I’ve started integrating; we should be able to land the rest including the IM pluginization for 1.0.

Commit summary:

XMPP queued output & initial retooling of DB queue manager to support non-Notice objects.

Queue handlers for XMPP individual & firehose output now send their XML stanzas to another output queue instead of connecting directly to the chat server. This lets us have as many general processing threads as we need, while all actual XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:

  • multiple windows shown in some chat clients (psi, gajim, kopete)
  • extra load on server
  • incoming message delivery forwarding issues

Database changes:

  • queue_item drops ‘notice_id’ in favor of a ‘frame’ blob. This is based on Craig Andrews’ work branch to generalize queues to take any object, but conservatively leaving out the serialization for now. Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:

  • Queue handlers should now define a handle() method instead of handle_notice()
  • QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning thread management (RespawningDaemon) infrastructure.
  • The polling XmppConfirmManager has been dropped, as the message is queued directly when saving IM settings.
  • Enable $config[‘queue’][‘debug_memory’] to output current memory usage at each run through the event loop to watch for memory leaks

To do:

  • Adapt XMPP i/o to component connection mode for multi-site support.
  • XMPP input can also be broken out to a queue, which would allow the actual notice save etc to be handled by general queue threads.
  • Make sure there are no problems with simply pushing serialized Notice objects to queues.
  • Find a way to improve interactive performance of the database-backed queue handler; polling is pretty painful to XMPP.
  • Possibly redo the way QueueHandlers are injected into a QueueManager. The grouping used to split out the XMPP output queue is a bit awkward.

StatusNet queue refactoring landed

Woohoo! After a couple months off and on adjusting the architecture to something that seems to meet our needs, I’ve merged my refactoring of StatusNet’s background queue processing to 0.9.x.

Some of my design notes are up on the wiki, with a couple updates based on tweaks I made from my original plans.

Key items from the commit summary:

Major refactoring of queue handlers to support running multiple sites in one daemon.

Key changes:

  • Initialization code moved from common.php to StatusNet class; can now switch configurations during runtime.
  • As a consequence, configuration files must now be idempotent… Be careful with constant, function or class definitions.
  • Control structure for daemons/QueueManager/QueueHandler has been refactored; the run loop is now managed by IoMaster run via scripts/queuedaemon.php IoManager subclasses are woken to handle socket input or polling, and may cover multiple sites.
  • Plugins can implement notice queue handlers more easily by registering a QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

  • This replaces most of the old *handler.php scripts; they’ve been refactored to the bare handler classes.
  • Spawns multiple child processes to spread load; defaults to CPU count on Linux and Mac OS X systems, or override with –threads=N
  • When multithreaded, child processes are automatically respawned on failure.
  • Threads gracefully shut down and restart when passing a soft memory limit (defaults to 90% of memory_limit), limiting damage from memory leaks.
  • Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:

QueueDaemon -> IoMaster -> IoManager
QueueManager [listen or poll] ->  QueueHandler
XmppManager [ping&  keepalive]
XmppConfirmManager [poll updates]

Todo:

  • Respawning features not currently available running single-threaded.
  • When running single-site, configuration changes aren’t picked up.
  • New sites or config changes affecting queue subscriptions are not yet handled without a daemon restart.
  • SNMP monitoring output to integrate with general tools (nagios, ganglia)
  • Convert XMPP confirmation message sends to use stomp queue instead of polling
  • Convert xmppdaemon.php to IoManager?
  • Convert Twitter status, friends import polling daemons to IoManager
  • Clean up some error reporting and failure modes
  • May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch.

Time zones: do we really need them?

While looking at StatusNet’s preferences dialog, I noticed we have a fairly generic but unfriendly timezone selector, done as a drop-down box with a raw list of zone names — this makes you find and select something like “America/Los_Angeles” or “Asia/Tokyo”:

Not only is the list very long and confusingly sorted, but the names are all either English or acronyms, making it hard to internationalize.

It occurs to me that we very rarely actually output formatted dates in the web interface anyway… most of the time we output relative times like “a few seconds ago” or “3 months ago”.

In fact, about the only place I can see that we’re outputting a fully formatted date is on an individual notice like this:

“Brion Vibber (brionv) ‘s status on Friday, 27-Nov-09 19:19:11 UTC”
“Brion Vibber (brionv) ‘s status on Friday, 27-Nov-09 11:19:11 PST”
etc

though we also have tooltips on the approximation eg “about a month ago”: “2009-11-27T11:19:11-08:00”

My own inclination would be to drop the timezone preferences entirely; in the rare cases where we output a formatted local date we can let client-side JavaScript do the actual date formatting with the client system’s actual time zone and language settings, with a UTC/GMT fallback for clients with JS disabled.

This drops an unnecessary and hard-to-select option field from preferences and site administration panels, and avoids inconsistencies when you travel or move and forget to update the timezone.

Any objections or suggestions for further refinement?

Location fuzzing in StatusNet

I had a chat today w/ Craig Andrews & Chris Vollick about StatusNet’s geolocation support and how we might be able to better present how location info will be used, to avoid freaking people out with unexpected maps to their homes. :)

Some notes and this mockup for expanded location setup in preferences are on the wiki.

The main points are:

  • provide context for the browser’s location lookup opt-in (don’t trigger it until you’re poking at your location options)
  • show a sample map in the prefs so you know what other people will see
  • provide a one-click way to disable browser location lookups entirely
  • provide a one-click way to reduce the accuracy to city-level, probably by default

Any thoughts?