Tag Archives: atompub

Is XMPP for the web ready?

A discussion started recently on the Jabber social mailing-list about the current state of XMPP support at Facebook. If you don’t remember the idea of a XMPP network for Facebook I’d say it’s rather normal considering they talked about it back in May 2008 and nothing has really happened since then. The discussion has quickly drifted toward the difficulty XMPP knows to really make it big and you can sense some despair within the community.

It seems there are several reasons for this.

First and foremost, XMPP is rather poorly integrated within browsers. Today you have to be part of the browser if you want to succeed. Perhaps the response will come from the rather brilliant lib Strophe. Strophe uses BOSH to connect to XMPP services since Javascript doesn’t offer much access to the low level socket connection object. I hope in the future browsers will offer a built-in XMPP API that can be accessed from Javascript allowing to avoid using BOSH.

Second of all, we need big players to start accepting to let go and start entering Jabber federations so that interoperability actually works. I assume companies have yet to find a way on how they can exercise control over the data they might expose through XMPP whilst finding a way to monetize them.

Third I’d say there is a critical lack of PR around XMPP as an added value to the business. To be honest there is the same issue with AtomPub in my opinion. Both are fantastic technologies but they don’t sell by themselves and I find there is a lack of support from companies to support them. SOAP might have been a crap technology but it was backed up by large businesses that were competing with each other (for the worse one might argue).

Finally, and to me this is the most critical reason, XMPP doesn’t integrate well with the web in general. From the browser side as I suggested above to the fact that it’s rather hard to combine web application with jabber ones. Its definitely doable but requires some careful attention to your architecture. Jack Moffitt argues it’s not XMPP’s fault. It’s quite true as a protocol but I believe it’s not really the right way to invite web developers to push it one step further if you blame them for doing it wrong.

I believe XMPP has tremendous potential but it’s still has some way to go before it finds its place.

amplee 0.6.1

I’ve just released amplee 0.6.1.
This release is a minor release that fixes a few annoying defects and improves overall performances of the internal of amplee:

  • removes many of the usage of the copy.copy function (thanks to Mohanara Gopala Krishnan for his previous help)
  • drops the cmemcache module in favor the python-memcached one
  • adds a prune(member) method to the indexer API
  • extends with a few helpful functions the atompub API
  • adds more unit tests
  • drops the graph subpackage from the egg distribution (and I will remove it altogether for license issue in the next release)

If you’re using 0.6.0 I would advise to upgrade. The API is fully compatible and you’ll experience a more stable product.

I really wish to thank Mohan for his feedback, patience and will to send patches.

I will release a next version to clean out some remaining issue and than I will move toward amplee 0.7 that’ll support Amara 2.0 and will see some internal refactoring.

AtomPub data model for music metadata storage and indexing

Noah Gift described a cool project he had started recently named Pylesystem. The idea was to monitor the filesystem for events regarding the creation and deletion of files, more specifically media files like music audio files. Noah explained he wanted to provide a simple metadata storage so that he could query it for getting information about his media library. He decided to use SQLAlchemy to abstract the storage as well as the query mechanism as much as possible whilst providing an easy and high level query interface.

I thought it’d be cool to give it a go using the AtomPub data model to store metadata. I therefore hacked a small example within a few hours this weekend using amplee.

I was quickly able to associate a collection per artist and album and a member per track. I used hachoir-metadata to get a few basic metadata about tracks as well as a couple of simple queries to the Musicbrainz web service to get more advanced information like tags, related URLs, etc.
Finally I used the basic indexing mechanism provided in amplee to perform simple query on the AtomPub store and then generate atom feeds on the fly based on the query result.
Overall it was really fun and I’m happy that it shows how to use AtomPub and amplee in a context that is not directly related to the web. The source code is of course available here.

Note that if you wish to run the code you’ll need the latest trunk of amplee. Otherwise you’ll need the current stable release of amara, hachoir-metadata and pyinotify.

XMPP, AtomPub and microblogging

Following my previous posts about my work on mixing AtomPub and XMPP together in a single application I’ve worked yesterday on the basic workflow of the application I write.

The first use case is to create a profile using OpenID.

  1. When landing on the application main page the user can enter his (or her) OpenID which will redirect him to his OpenID provider for validation.
  2. Once validated and accepted the user comes back to the application which shows a simple profile page pre-filled with information using the simple registration extension of OpenID.
  3. Upon submitting his profile for registration the application stores the profile following the format described in XEP-0154. It also creates a node in a Jabber PubSub service that will be used as the top-level node of the user. It finally creates a workspace specific to the user within the AtomPub service.

Note the the application I’m writing will conflate two similar notions into one and make them context free:

  1. node in PubSub
  2. collection in AtomPub

Both terms will refer to a channel in the application. So sub-nodes of the user top-level node will be called a channels and each collection within the user workspace will also be called a channel. In other words publishing data to a channel means publishing data to both the PubSub node and the AtomPub collection at the same time no matter the protocol chosen to perform the operation. The channel convention is arbitrary but helpful as it hides the underlying protocol away.

Once a user is registered to the application he can log in using his OpenID. After logging in the user can register, start and stop the internal XMPP client associated with his account. The first step is to register it from the application. Once registered the client can be started and stopped at will.
When the internal XMPP client is registered the user ought to use his favorite Jabber client to register a new account to the Jabber server. Then he should subscribe to the internal client’s contact list that will automatically accept it (probably in a better scenario subscription should be moderated from the user’s profile page).

The second use case is to create channels

  1. The user could create channels using either his Jabber client or using the AtomPub interface.
  2. In both cases the application would create both the according PubSub sub-node to the user’s top-level node and the appropriate AtomPub collection.

The third use case is to publish data to channels with XMPP

  1. The user uses his own Jabber client to send a message to the internal jabber client like this: “publish channel Hello world”. This would translate into “Publish ‘Hello world’ to channel” where channel is the name of the channel to publish to.
  2. The jabber server would forward this message to the internal client who upon parsing it would understand it must publish the sent data to the according PubSub node. First it would enclose the data into the atom:content of an atom entry and publish that entry as the node’s item.
  3. When the item is published the Jabber server would send a notification back to the internal client which could then create and store the according AtomPub member entry to the appropriate collection. The entry would become visible via the Atom feed of the collection.

Note that waiting for the notification to be propagated back to store the atom entry within the collection rather than doing it immediately when publishing to the PubSub node is not gratuitous. Indeed the user could publish items directly from an external client that understands PubSub. By doing it the way described above we ensure that even in such a case the internal client will be informed a new item was published to a node it is subscribed to and therefore the application will keep in sync’ no matter what.
The fourth use case is to publish data to channels with AtomPub
This would be similar to the previous use case except the entry point would be the AtomPub interface and that of course a message would be sent through XMPP accordingly.
Deleting items from channels would work in the same fashion.
Of course the user’s channels would then be publicly available and other users could subscribe to the Atom feed, to the internal XMPP client or to the user’s PubSub nodes.

From there on the application could be extended so that for instance users can comment to each other and ensure that the channels’ items contain that information, for instance using RFC 4685 within each item.

These are few things I’ve been working on. The application is not ready yet and it might take a few days for it to complete and hopefully some of you will be interested in testing it then.
Regarding the platform used, Python and the following products:

  1. headstock through Kamaelia for the XMPP layer
  2. amplee for the AtomPub interface
  3. CherryPy 3.1 for the HTTP serving
  4. amara for the XML handling

XMPP, AtomPub, headstock, amplee and the BBC

Matthew Wood, from the BBC Radio Labs, posted a few days ago an exciting note regarding fun he was having with XMPP and services such as last.fm to inform user, via XMPP messages, of BBC radios broadcasting music they might like based on thier last.fm profile. I thought this was fantastic but I was even more excited when I read another note where he explained that he was using PubSub as a mean to carry and distribute metadata about BBC shows using Atom entries as the metadata format.
This evening I spent three hours expanding on the simple chat example coming with headstock to talk with the PubSub service. Then I integrated amplee as a way to offer an AtomPub interface at the same time. This means that when the demo starts both a XMPP client, connecting to Matt’s service, and an AtomPub server, using amplee and served by CherryPy, are started. The XMPP client asks the server about PubSub nodes. For each node representing BBC channels I create an atompub collection within its own workspace. Simultaneously I subscribe to those nodes. I then ask the XMPP server for items belonging to those nodes and for each item, representing metadata about a show for instance, I create an atom entry that I store within the AtomPub store.
This means one can then simply subscribe with a feed reader to a given collection and/or a XMPP PubSub node. All of this happening on the fly starting from an empty AtomPub service document.
Eventually I will add support so that when an Atom entry is POSTed to a AtomPub collection, the according PubSub stanza is pushed towards the service (I doubt Matt’s service accepts it though) allowing for microblogging support.
The source code of the example can be found here. If you want to understand how it works you might want to read this quick word I wrote about Kamaelia first which is at the core of headstock.

AtomPub and XMPP

There has been quite a lot of discussions around the use of XMPP in a web context and how to blend HTTP and XMPP protocols for a better social network experience.

I’ve been working on implementing XMPP for a while now on my spare time and even though it’s been a much slower effort I would have liked, I’ve been able to have a fantastic fun with it. Most recently I’ve started integrating XMPP PubSub along with AtomPub using my AtomPub implementation, amplee.
The idea is to use amplee as an AtomPub library from within XMPP PubSub handlers. For instance I map AtomPub collections to PubSub nodes . When an atom entry is published to a node and that the client receives an acknowledgment of that publication, the handler uses amplee to store the entry in the AtomPub collection as well, respecting of course the RFC specification. Similarly if an item is deleted, the handler uses amplee to remove the entry from the collection. This works both ways, we can also run an AtomPub web service which issues the right XMPP stanzas according to the operation carried. So a POST on a collection would publish the atom entry to the XMPP server. The web service would obviously expose the Atom feed of the collection.
This is just a stub, but the idea here is to associate both protocols so that they cooperate to expand the audience of your network. It’d be easy to consider allowing people commenting to a blog entry using their XMPP client. You blog entry would for instance advert the XMPP service and node name where to publish items. The user could then subscribe to than node and publish items.
That’s one reason why I had written amplee as an AtomPub library that could work outside of the HTTP protocol. RFC 5023 defines the protocol using HTTP but the root idea behind it works well in the context of XMPP and maybe AtomPub is the protocol that will rule them all.