Category: StatusNet

Mirrored from my entry on StatusNet company blog

StatusNet 0.9.6 is ready to download:

http://status.net/download
http://status.net/wiki/StatusNet_0.9.6

This version is now live on status.net hosted sites and identi.ca, and is a recommended upgrade with bug fixes and improved client authentication.

OAuth authentication for the Twitter-compatible API has been updated significantly, which may require some client apps to change their behavior for new users to sign in:

• Updated to OAuth 1.0a — we require you to use the verifier, and 1.0 wont work
• A new “oob” pin-based workflow for apps with limited web capability
• Tokens exchange and authorization happen over SSL by default
• A New mode=desktop parameter for apps displaying a stripped down, “lite” version of the authorization page in a webview
• An anonymous consumer for apps that don’t want to register for a custom consumer key and secret (good for apps that need to work with multiple StatusNet instances)
• When a user declines to authorize a request token, we notify the client by calling the verified callback with the oauth_problem=user_refused parameter

Please test your client apps to confirm they still function; even if existing access tokens work, be sure to try signing up a new user too! If you have troubles, ping Zach Copley who’s become our resident OAuth wizard, either directly via StatusNet or in our IRC channel on FreeNode.

Changes from 0.9.6 release candidate 1:

• fix for broken group pages when logged out
• fix for stuck ping queue entries when bad profile
• fix for bogus single-user nickname config entry error
• i18n updates
• nofollow updates
• SSL-only mode secure cookie fix
• experimental ApiLogger plugin for usage data gathering
• experimental follow-everyone plugin

Other notable changes this version:

• Site moderators can now delete groups.
• New themes: clean, shiny, mnml, victorian
• New YammerImport plugin allows site admins to import non-private profiles and message from an authenticated Yammer site.
• New experimental plugins: AnonFavorites, SlicedFavorites, GroupFavorited, ForceGroup, ShareNotice
• OAuth upgraded to 1.0a
• Localization updates now include plugins, thanks to translatewiki.net!
• SSL link generation should be more consistent; alternate SSL URLs can be set in the admin UI for more parts of the system.
• Experimental backupuser.php, restoreuser.php command-line scripts to dump/restore a user’s complete activity stream. Can be used to transfer accounts manually between sites, or to save a backup before deleting.
• Unicode fixes for OStatus notices
• Header metadata on notice pages to aid in manual reposting on Facebook
• Lots of little fixes…

A complete list of changes since 0.9.5 is included in the ‘Changelog’ file in the RC download and on the wiki:
http://status.net/wiki/StatusNet_0.9.6/Changelog

— brion

Mirrored from my entry on StatusNet company blog

StatusNet 0.9.6 release candidate 1 is ready to download: http://statusnetdev.net/rc/statusnet-0.9.6rc1.tar.gz

We’ll be pushing this up to status.net hosted sites and identi.ca, and releasing the general 0.9.6 download, later this week.

OAuth authentication for the Twitter-compatible API has been updated significantly, which may require some client apps to change their behavior for new users to sign in:

1. updated to 1.0a – we require you to use the verifier, and 1.0 wont work
2. new “oob” pin-based workflow for apps with limited web capability
3. tokens exchange and authorization happen over SSL by default,
4. new mode=desktop parameter for apps displaying a stripped down, “lite” version authorization page in a webview
5. an anonymous consumer for apps that don’t want to register for a custom consumer key and secret (good for apps that need to work with multiple StatusNet instances

When a user declines to authorize a request token, we notify the client by calling the verified callback with the oauth_problem=user_refused parameter

Please test your client apps to confirm they still function; even if existing access tokens work, be sure to try signing up a new user too! If you have troubles, ping Zach Copley who’s become our resident OAuth wizard, either directly via StatusNet or in our IRC channel on FreeNode.

Other notable changes this version:

• Site moderators can now delete groups.
• New themes: clean, shiny, mnml, victorian
• New YammerImport plugin allows site admins to import non-private profiles and message from an authenticated Yammer site.
• New experimental plugins: AnonFavorites, SlicedFavorites, GroupFavorited, ForceGroup, ShareNotice
• OAuth upgraded to 1.0a
• Localization updates now include plugins, thanks to translatewiki.net!
• SSL link generation should be more consistent; alternate SSL URLs can be set in the admin UI for more parts of the system.
• Experimental backupuser.php, restoreuser.php command-line scripts to dump/restore a user’s complete activity stream. Can be used to transfer accounts manually between sites, or to save a backup before deleting.
• Unicode fixes for OStatus notices
• Header metadata on notice pages to aid in manual reposting on Facebook
• Lots of little fixes…

A complete list of changes since 0.9.5 is included in the ‘Changelog’ file in the RC download.

— brion

While you’re all waiting for the StatusNet 0.9.6 patch release, here’s an update on our StatusNet 1.0.x infrastructure projects…

To make upgrades and plugin installation go more smoothly, we’re overhauling the schema API to let us use a flexible, database-independent table definition language for the entire database. For now I’ve been adapting the definition syntax from Drupal’s schema API though we’re not (currently) actually using any of their code. (It looks like the GPL licensing should be clear, so it’s a matter of whether the code would be cleanly sharable.)

Work is progressing on the schema-x branch on my StatusNet git clone, and will be folded into mainline 1.0.x once it’s reasonably stable.

So how does all this work, you ask?

Let’s take a peek at the foreign_subscription table. This is used to record known subscription/friends relationships on other services, such as when you’ve connected a Twitter or Facebook account to your StatusNet setup.

It’s a fairly simple relational database table, which connects two records from another table (foreign_user), identified by a service ID key and a per-service user ID for each connected record. The table definition, using the Drupal-style definition arrays, looks like this:

$schema['foreign_subscription'] = array(
    'fields' => array(
        'service' => array('type' => 'int', 'not null' => true, 'description' => 'service where relationship happens'),
        'subscriber' => array('type' => 'int', 'size' => 'big', 'not null' => true, 'description' => 'subscriber on foreign service'),
        'subscribed' => array('type' => 'int', 'size' => 'big', 'not null' => true, 'description' => 'subscribed user'),
        'created' => array('type' => 'datetime', 'not null' => true, 'description' => 'date this record was created'),
    ),
    'primary key' => array('service', 'subscriber', 'subscribed'),
    'foreign keys' => array(
        'foreign_subscription_service_fkey' => array('foreign_service', array('service' => 'id')),
        'foreign_subscription_subscriber_fkey' => array('foreign_user', array('subscriber' => 'id', 'service' => 'service')),
        'foreign_subscription_subscribed_fkey' => array('foreign_user', array('subscribed' => 'id', 'service' => 'service')),
    ),
    'indexes' => array(
        'foreign_subscription_subscriber_idx' => array('service', 'subscriber'),
        'foreign_subscription_subscribed_idx' => array('service', 'subscribed'),
    ),
);

That DB-independent schema definition gets filtered and converted to the particular data types that each database engine will support.

Using the dumpschema.php test script, we can have the Schema system peek at my 0.9.x-installed MySQL database and pull the actual current table and index definitions:

$ php scripts/dumpschema.php foreign_subscription
'foreign_subscription' => array(
    'fields' => array(
        'service' => array('type' => 'int', 'not null' => true, 'description' => 'service where relationship happens'),
        'subscriber' => array('type' => 'int', 'not null' => true, 'description' => 'subscriber on foreign service'),
        'subscribed' => array('type' => 'int', 'not null' => true, 'description' => 'subscribed user'),
        'created' => array('type' => 'datetime', 'not null' => true, 'description' => 'date this record was created')
    ),
    'primary key' => array('service', 'subscriber', 'subscribed'),
    'indexes' => array(
        'foreign_subscription_subscriber_idx' => array('subscriber'),
        'foreign_subscription_subscribed_idx' => array('subscribed')
    )
)

To be able to update existing database tables, it needs to calculate the differences between what’s in the DB now, and what it’s supposed to look like.

We can pass the --diff option to dumpschema.php to visualize this:

$ php scripts/dumpschema.php --diff foreign_subscription
'foreign_subscription' => array(
    'fields' => array(
        'OLD subscriber' => array('type' => 'int', 'not null' => true, 'description' => 'subscriber on foreign service'),
        'NEW subscriber' => array('type' => 'bigint', 'not null' => true, 'description' => 'subscriber on foreign service'),
        'OLD subscribed' => array('type' => 'int', 'not null' => true, 'description' => 'subscribed user'),
        'NEW subscribed' => array('type' => 'bigint', 'not null' => true, 'description' => 'subscribed user')
    ),
    'indexes' => array(
        'OLD foreign_subscription_subscriber_idx' => array('subscriber'),
        'NEW foreign_subscription_subscriber_idx' => array('service', 'subscriber'),
        'OLD foreign_subscription_subscribed_idx' => array('subscribed'),
        'NEW foreign_subscription_subscribed_idx' => array('service', 'subscribed')
    )
)

We can see that the foreign_subscription table’s definition has been updated with some fixes: the user ID field types have been changed to match the referenced foreign_user table, and indexes have been tweaked to take into account that foreign user IDs need to be looked up within the context of a service ID.

Neat, right? But not all that useful yet. What we really want is for it to be able to produce SQL statements to modify the table to the state it should be:

$ php scripts/dumpschema.php --update foreign_subscription
-- 
-- foreign_subscription
-- 
DROP INDEX foreign_subscription_subscriber_idx ON foreign_subscription;
DROP INDEX foreign_subscription_subscribed_idx ON foreign_subscription;
ALTER TABLE foreign_subscription MODIFY COLUMN subscriber bigint not null comment 'subscriber on foreign service',
MODIFY COLUMN subscribed bigint not null comment 'subscribed user';
CREATE INDEX foreign_subscription_subscriber_idx ON foreign_subscription (service,subscriber);
CREATE INDEX foreign_subscription_subscribed_idx ON foreign_subscription (service,subscribed);

These are the SQL statements that would be run during a ‘checkschema’ operation, modifying the existing table to match our new definition. Each database engine can override bits of the SQL generation; there are a number of differences between how MySQL and PostgreSQL do things for instance.

If we’re super-brave, we can go further and enable foreign key support, which makes the relationships between tables explicit and lets the database raise errors if we try to do something that would create inconsistent data. Our statusnet.sql table definitions file has had ‘REFERENCES’ clauses for a long time, but the way we declared them MySQL doesn’t actually enforce them — they’re really just for documentation purposes. Our PostgreSQL definitions have had actual functioning foreign keys, but haven’t gotten as much testing as the MySQL version.

Enabling foreign key support for MySQL can help developers and testers to confirm correct behavior, without forcing them to also be on in production where the performance premium might be too high. (In an ideal world though I’d love to have them running in production too!)

To turn on this option, add to config.php:

  $config['db']['mysql_foreign_keys'] = true;

With foreign keys enabled, the schema system will happily try to add the key defs:

$ php scripts/dumpschema.php --update foreign_subscription
-- 
-- foreign_subscription
-- 
DROP INDEX foreign_subscription_subscriber_idx ON foreign_subscription;
DROP INDEX foreign_subscription_subscribed_idx ON foreign_subscription;
ALTER TABLE foreign_subscription MODIFY COLUMN subscriber bigint not null comment 'subscriber on foreign service',
MODIFY COLUMN subscribed bigint not null comment 'subscribed user',
ADD CONSTRAINT foreign_subscription_service_fkey FOREIGN KEY (service) REFERENCES foreign_service (id),
ADD CONSTRAINT foreign_subscription_subscriber_fkey FOREIGN KEY (subscriber,service) REFERENCES foreign_user (id,service),
ADD CONSTRAINT foreign_subscription_subscribed_fkey FOREIGN KEY (subscribed,service) REFERENCES foreign_user (id,service);
CREATE INDEX foreign_subscription_subscriber_idx ON foreign_subscription (service,subscriber);
CREATE INDEX foreign_subscription_subscribed_idx ON foreign_subscription (service,subscribed);

Note that things probably won’t all work 100% right with foreign keys on MySQL yet; for one thing they don’t work on MyISAM tables, which are needed to use the full-text search for notice and profile data. :)

But Brion, how do these tables get created in the first place with no SQL file?

The installer has been updated to use the Schema interface. The core (non-plugin) table definitions are listed in db/core.php, which are pulled to run through Schema for creation.

This has necessitated a little bit of code rearrangement to get everything Schema needs to work up and running mid-install, but it seems to more or less work. Note that there may be some minor issues sill keeping the installer from 100% working at the moment.

We saw how the logic works, but when do upgrades actually happen for core tables?

That hasn’t gone in just yet… my current plan is to fold in those checks with the rest of the checkschema event, so we’ll handle core and plugin tables in the same swath of stuff.

But won’t that be slow to check all those tables?

It would indeed be pretty slow to do a full check of what every table looks like on every hit. However, we should be able to eliminate that and still get nice auto-updating behavior.

This hasn’t yet been implemented, but I plan to add a ‘schema_version’ table listing all the known tables in the system and a checksum of their schema definition array as of when we last updated them. Now, instead of pulling metadata for dozens of tables (several queries each), we only need to do one quick lookup on the ‘schema_version’ table — which itself can be cached.

If a checksum is missing or doesn’t match the array we’ve got now, then we know we need to go check that table out in detail.

So Brion, how does this change how plugins add tables?

Existing plugins using the 0.9.x schema interface should actually still work: your old TableDef/ColumnDef stuff will be automatically converted into the new array layout when you pass it in to Schema->ensureTable().

But you’ll make your life nicer if you switch fully to the new system, since it’ll let us drop a lot of duplicated code that plugins have to do in 0.9…

If the internal API stays stable (and it may not ;), you would have your table’s class extend Managed_Data_Object, an extended version of our cachable Memcache_Data_Object which knows how to build its DB_DataObject internal metadata and Memcache_Data_Object cache keys from one of our table definition arrays. Add a getSchema static method to return the new-style array, and run that through Schema::ensureTable() in your onCheckSchema handler.

More of the common logic will get moved into the Plugin class, so once it’s finished you should be able to simply declare which table names/classes to add without messing with Schema manually.

Another long-forgotten feature… I’ve added group deletion for site moderators in current 0.9.x branch (not yet live on status.net hosted sites).

I think I got the most important caching loose ends tied up but wouldn’t mind a quick lookover!

• Basic UI
• delete related tables
• cache fixes

Existing group messages will remain in the system, but the group itself will cease to exist. (The !links in notices will now lead to an error page.) The group name and aliases are freed up and can be used again; creating a new group with the same name will not retain any group membership, etc.

Group memberships are deleted at the local level, but there’s not yet a way for the server to signal that the group is gone. As a result, remote OStatus subscribers may still be listed as members of the deleted group on their own sites until they leave it manually.

It should be possible to allow group admins to delete their own groups as well, with some more fiddling with the permissions system.

Hey all!

One of the other projects I’ve been getting back on track with lately is updating our database schema setup & updating system for StatusNet 1.0.x.

1.0’s going to have a lot of little DB tweaks like changing indexes and adding a few fields to make some of our bulk queries a lot faster. We’ve also broken out more features to plugins like XMPP support and its newer AIM, MSN, and IRC cousins.

I want to make sure that setup and update of StatusNet core and plugins works easily and consistently for developers and site maintainers, and especially for those running slightly different systems than ours (such as PostgreSQL installations, which have definitely seen some regressions.)

A few weeks ago I posted some initial notes on the wiki about limitations we’ve found in practice with the Schema API that we introduced last year for plugins, and examined the possibility of adapting Drupal’s schema definition layout structure.

This gives us additional flexibility without inventing our own data structures from thin air; it’s enough data to produce live tables and DB_DataObject’s metadata, and gives us a chance to bake in support for non-MySQL backends from the beginning.

I’d love for folks working with PostgreSQL (or with other experimental backends) to poke around at my ‘schema-x’ work branch and send feedback or fixes. Thoughts and ideas about slightly-funky stuff like binary blobs, enums, and db-specific fulltext indexing are especially welcome. :)

For 1.0.x the only top-tier supported backends will be MySQL and PostgreSQL, but we’d love to see more!

I’ve pulled in some fixes from one of my old experimental branches which gets StatusNet‘s bit.ly URL shortener plugin working again.

Some while ago, bit.ly changed to requiring an API key for all shortening requests, which broke our old BitlyUrlPlugin.

The updated version now in master & 0.9.x branches allows setting sitewide bit.ly credentials, either through the addPlugin() parameters or in the admin panels on the site:

If no credentials are set up, then bit.ly isn’t offered as a shortening service to end-users; if they are set up then the global keys are used for all bit.ly requests. Settings in the admin panel will override those in the addPlugin() parameters; this allows setting a default for a big hosted site farm like *.status.net, while individual sites can customize their settings to make use of the statistic tracking features or custom domains that bit.ly pro accounts offer.

It would be nice to further add a per-user override, but a little refactoring on the ‘other settings’ panel is needed to keep everything together in one place.

This isn’t available on status.net hosted sites just yet, but it should be coming soon!

A few quick StatusNet-related updates:

• 0.9.4 is about to be released (beta2 tag running on live servers now)
• StatusNet 0.9.4 is the first release to include an Esperanto translation of the UI — big thanks to all our translators at TranslateWiki.net who’ve helped with EO and so many other languages!
• We’re still chugging away at a StatusNet mobile client (open source, based on the Appcelerator Titanium cross-platform runtime) Probably betas for iPhone users next week? We’ll also want Android testers, but you don’t need to sign up. :)
• Among other things, I’m poking at our experimental TinyMCE-based wysiwyg edit plugin. More work is needed before it’s ready for general usage, but it’s kinda neat! (So much easier than adding wysiwig to a wiki since there’s no need to round-trip markup through edits over and over. ;)