XMPP, AtomPub and microblogging
May 20, 2008
Following my previous posts about my work on mixing AtomPub and XMPP together in a single application I've worked yesterday on the basic workflow of the application I write.
The first use case is to create a profile using OpenID.
1. When landing on the application main page the user can enter his (or her) OpenID which will redirect him to his OpenID provider for validation.
2. Once validated and accepted the user comes back to the application which shows a simple profile page pre-filled with information using the simple registration extension of OpenID.
3. Upon submitting his profile for registration the application stores the profile following the format described in XEP-0154. It also creates a node in a Jabber PubSub service that will be used as the top-level node of the user. It finally creates a workspace specific to the user within the AtomPub service.
Note the the application I'm writing will conflate two similar notions into one and make them context free:
1. node in PubSub
2. collection in AtomPub
Both terms will refer to a channel in the application. So sub-nodes of the user top-level node will be called a channels and each collection within the user workspace will also be called a channel. In other words publishing data to a channel means publishing data to both the PubSub node and the AtomPub collection at the same time no matter the protocol chosen to perform the operation. The channel convention is arbitrary but helpful as it hides the underlying protocol away.
Once a user is registered to the application he can log in using his OpenID. After logging in the user can register, start and stop the internal XMPP client associated with his account. The first step is to register it from the application. Once registered the client can be started and stopped at will.
When the internal XMPP client is registered the user ought to use his favorite Jabber client to register a new account to the Jabber server. Then he should subscribe to the internal client's contact list that will automatically accept it (probably in a better scenario subscription should be moderated from the user's profile page).
The second use case is to create channels
1. The user could create channels using either his Jabber client or using the AtomPub interface.
2. In both cases the application would create both the according PubSub sub-node to the user's top-level node and the appropriate AtomPub collection.
The third use case is to publish data to channels with XMPP
1. The user uses his own Jabber client to send a message to the internal jabber client like this: “publish channel Hello world”. This would translate into “Publish 'Hello world' to channel” where channel is the name of the channel to publish to.
2. The jabber server would forward this message to the internal client who upon parsing it would understand it must publish the sent data to the according PubSub node. First it would enclose the data into the atom:content of an atom entry and publish that entry as the node's item.
3. When the item is published the Jabber server would send a notification back to the internal client which could then create and store the according AtomPub member entry to the appropriate collection. The entry would become visible via the Atom feed of the collection.
Note that waiting for the notification to be propagated back to store the atom entry within the collection rather than doing it immediately when publishing to the PubSub node is not gratuitous. Indeed the user could publish items directly from an external client that understands PubSub. By doing it the way described above we ensure that even in such a case the internal client will be informed a new item was published to a node it is subscribed to and therefore the application will keep in sync' no matter what.
The fourth use case is to publish data to channels with AtomPub
This would be similar to the previous use case except the entry point would be the AtomPub interface and that of course a message would be sent through XMPP accordingly.
Deleting items from channels would work in the same fashion.
Of course the user's channels would then be publicly available and other users could subscribe to the Atom feed, to the internal XMPP client or to the user's PubSub nodes.
From there on the application could be extended so that for instance users can comment to each other and ensure that the channels' items contain that information, for instance using RFC 4685 within each item.
These are few things I've been working on. The application is not ready yet and it might take a few days for it to complete and hopefully some of you will be interested in testing it then.
Regarding the platform used, Python and the following products:
1. headstock through Kamaelia for the XMPP layer
2. amplee for the AtomPub interface
3. CherryPy 3.1 for the HTTP serving
4. amara for the XML handling
The first use case is to create a profile using OpenID.
1. When landing on the application main page the user can enter his (or her) OpenID which will redirect him to his OpenID provider for validation.
2. Once validated and accepted the user comes back to the application which shows a simple profile page pre-filled with information using the simple registration extension of OpenID.
3. Upon submitting his profile for registration the application stores the profile following the format described in XEP-0154. It also creates a node in a Jabber PubSub service that will be used as the top-level node of the user. It finally creates a workspace specific to the user within the AtomPub service.
Note the the application I'm writing will conflate two similar notions into one and make them context free:
1. node in PubSub
2. collection in AtomPub
Both terms will refer to a channel in the application. So sub-nodes of the user top-level node will be called a channels and each collection within the user workspace will also be called a channel. In other words publishing data to a channel means publishing data to both the PubSub node and the AtomPub collection at the same time no matter the protocol chosen to perform the operation. The channel convention is arbitrary but helpful as it hides the underlying protocol away.
Once a user is registered to the application he can log in using his OpenID. After logging in the user can register, start and stop the internal XMPP client associated with his account. The first step is to register it from the application. Once registered the client can be started and stopped at will.
When the internal XMPP client is registered the user ought to use his favorite Jabber client to register a new account to the Jabber server. Then he should subscribe to the internal client's contact list that will automatically accept it (probably in a better scenario subscription should be moderated from the user's profile page).
The second use case is to create channels
1. The user could create channels using either his Jabber client or using the AtomPub interface.
2. In both cases the application would create both the according PubSub sub-node to the user's top-level node and the appropriate AtomPub collection.
The third use case is to publish data to channels with XMPP
1. The user uses his own Jabber client to send a message to the internal jabber client like this: “publish channel Hello world”. This would translate into “Publish 'Hello world' to channel” where channel is the name of the channel to publish to.
2. The jabber server would forward this message to the internal client who upon parsing it would understand it must publish the sent data to the according PubSub node. First it would enclose the data into the atom:content of an atom entry and publish that entry as the node's item.
3. When the item is published the Jabber server would send a notification back to the internal client which could then create and store the according AtomPub member entry to the appropriate collection. The entry would become visible via the Atom feed of the collection.
Note that waiting for the notification to be propagated back to store the atom entry within the collection rather than doing it immediately when publishing to the PubSub node is not gratuitous. Indeed the user could publish items directly from an external client that understands PubSub. By doing it the way described above we ensure that even in such a case the internal client will be informed a new item was published to a node it is subscribed to and therefore the application will keep in sync' no matter what.
The fourth use case is to publish data to channels with AtomPub
This would be similar to the previous use case except the entry point would be the AtomPub interface and that of course a message would be sent through XMPP accordingly.
Deleting items from channels would work in the same fashion.
Of course the user's channels would then be publicly available and other users could subscribe to the Atom feed, to the internal XMPP client or to the user's PubSub nodes.
From there on the application could be extended so that for instance users can comment to each other and ensure that the channels' items contain that information, for instance using RFC 4685 within each item.
These are few things I've been working on. The application is not ready yet and it might take a few days for it to complete and hopefully some of you will be interested in testing it then.
Regarding the platform used, Python and the following products:
1. headstock through Kamaelia for the XMPP layer
2. amplee for the AtomPub interface
3. CherryPy 3.1 for the HTTP serving
4. amara for the XML handling
Facebook goes XMPP
May 14, 2008
Via Peter Saint-André we learn that facebook is adding XMPP as a mean to connect and use its services. Independently from what usage is made of Facebook this is a great news for the Jabber community in general and the protocol itself as it demonstrates that, unlike some nay-sayers used to claim in the past, XMPP will be the the true sibling to HTTP when it comes to social networking and delivering near real-time data on a large scale. Indeed, considering that media companies like the BBC, are looking at using too (even if it's purely at a research level for now) it seems to me that XMPP has great years ahead.
I find that interesting though that Google, which has had a Jabber network for years now, has never been able to actually push much that way. I mean Orkut and Google Talk have been able to communicate for quite some time and yet it seems the Facebook prospect is more exciting for the Jabber community than Google's usage ever was. Maybe it comes down to the fact Orkut is a pale shadow of Facebook.
On a personal level, I won't complain that Google doesn't push advertisement through Google Talk of course but one may wonder why it doesn't do so from a business point of view. well probably they know having ads that way would drive their customers away, as they say (emphasis mine):
We'll see how Facebook handles it considering the big fiasco Beacon was.
As a developer I've long felt frustrated at how limited the Google Talk standalone application is. Google Talk gadget is a bit better featured but it pains me having to use it when I prefer its big brother. Still both are very limited. To be fair though most well known IM clients speaking XMPP are limited in regards to what the protocol and its extensions offer. It's sad so little support PubSub and even those that do are somewhat basic in their support.
Let's hope we'll more and more XMPP applications out of the context of instant messaging or with a larger scope like microblogging.
I find that interesting though that Google, which has had a Jabber network for years now, has never been able to actually push much that way. I mean Orkut and Google Talk have been able to communicate for quite some time and yet it seems the Facebook prospect is more exciting for the Jabber community than Google's usage ever was. Maybe it comes down to the fact Orkut is a pale shadow of Facebook.
On a personal level, I won't complain that Google doesn't push advertisement through Google Talk of course but one may wonder why it doesn't do so from a business point of view. well probably they know having ads that way would drive their customers away, as they say (emphasis mine):
There are no ads in your chat sessions or your Quick Contacts list. Once a chat is saved, however, it becomes just like a Gmail message. And just as you may see relevant ads next to your Gmail messages, there now may be ads alongside your saved chats. Ads are only displayed when you're viewing a saved chat, and as with all ads in Gmail, they are matched entirely by computers. Only ads classified as Family-Safe are shown and we are constantly improving our technologies to prevent displaying any inappropriate ads. One of the things many Gmail users have told us is how much they appreciate the unobtrusive text ads in Gmail, as opposed to the large, irrelevant, blinking banner ads they often see in other services, and many have even cited the usefulness of the ads in Gmail.
We'll see how Facebook handles it considering the big fiasco Beacon was.
As a developer I've long felt frustrated at how limited the Google Talk standalone application is. Google Talk gadget is a bit better featured but it pains me having to use it when I prefer its big brother. Still both are very limited. To be fair though most well known IM clients speaking XMPP are limited in regards to what the protocol and its extensions offer. It's sad so little support PubSub and even those that do are somewhat basic in their support.
Let's hope we'll more and more XMPP applications out of the context of instant messaging or with a larger scope like microblogging.
XMPP, AtomPub, headstock, amplee ...
May 13, 2008
Matthew Wood, from the BBC Radio Labs, posted a few days ago an exciting note regarding fun he was having with XMPP and services such as last.fm to inform user, via XMPP messages, of BBC radios broadcasting music they might like based on thier last.fm profile. I thought this was fantastic but I was even more excited when I read another note where he explained that he was using PubSub as a mean to carry and distribute metadata about BBC shows using Atom entries as the metadata format.
This evening I spent three hours expanding on the simple chat example coming with headstock to talk with the PubSub service. Then I integrated amplee as a way to offer an AtomPub interface at the same time. This means that when the demo starts both a XMPP client, connecting to Matt's service, and an AtomPub server, using amplee and served by CherryPy, are started. The XMPP client asks the server about PubSub nodes. For each node representing BBC channels I create an atompub collection within its own workspace. Simultaneously I subscribe to those nodes. I then ask the XMPP server for items belonging to those nodes and for each item, representing metadata about a show for instance, I create an atom entry that I store within the AtomPub store.
This means one can then simply subscribe with a feed reader to a given collection and/or a XMPP PubSub node. All of this happening on the fly starting from an empty AtomPub service document.
Eventually I will add support so that when an Atom entry is POSTed to a AtomPub collection, the according PubSub stanza is pushed towards the service (I doubt Matt's service accepts it though) allowing for microblogging support.
The source code of the example can be found here. If you want to understand how it works you might want to read this quick word I wrote about Kamaelia first which is at the core of headstock.
This evening I spent three hours expanding on the simple chat example coming with headstock to talk with the PubSub service. Then I integrated amplee as a way to offer an AtomPub interface at the same time. This means that when the demo starts both a XMPP client, connecting to Matt's service, and an AtomPub server, using amplee and served by CherryPy, are started. The XMPP client asks the server about PubSub nodes. For each node representing BBC channels I create an atompub collection within its own workspace. Simultaneously I subscribe to those nodes. I then ask the XMPP server for items belonging to those nodes and for each item, representing metadata about a show for instance, I create an atom entry that I store within the AtomPub store.
This means one can then simply subscribe with a feed reader to a given collection and/or a XMPP PubSub node. All of this happening on the fly starting from an empty AtomPub service document.
Eventually I will add support so that when an Atom entry is POSTed to a AtomPub collection, the according PubSub stanza is pushed towards the service (I doubt Matt's service accepts it though) allowing for microblogging support.
The source code of the example can be found here. If you want to understand how it works you might want to read this quick word I wrote about Kamaelia first which is at the core of headstock.
headstock: XMPP library based on ...
May 12, 2008
I'm pleased to announce the release of the first version of headstock, my XMPP implementation using the Kamaelia library. My main motivation behind working on that project rather than using one of the existing libraries was that I wanted to find a challenging project to work with Kamaelia. XMPP seemed challenging enough.
This first release is flagged as beta. Not production ready at all but offers enough so that you can play with it and actually use it for integrating XMPP into your applications. Documentation and tests are badly missing and they will eventually come but for now you'll have to go through the code. I know this is not attractive but I though it was worth to be released nonetheless. Enjoy.
This first release is flagged as beta. Not production ready at all but offers enough so that you can play with it and actually use it for integrating XMPP into your applications. Documentation and tests are badly missing and they will eventually come but for now you'll have to go through the code. I know this is not attractive but I though it was worth to be released nonetheless. Enjoy.
French O'Reilly has shutdown
May 09, 2008
A sad news for French speakers as the French O'Reilly editor has shutdown completely due to the lack of sales. They'll be missed.
Redmine vs Trac
May 07, 2008
Following my last post in regards to a move from subversion to mercurial I received many interesting comments. One key aspect that those comments showed was that no matter which DCVS I would use it would be critical to me that it could be interfaced with Trac since that's the tool I'm using to manage my projects. Jim Jones hinted that I could also see the problem the other way around and decide to change for a different software management tool that would be better at handling the DCVS I'd choose. He led me to discover Redmine.
I wasn't very motivated by the idea of migrating from Trac to a different tool. Many reasons to that:
1. Trac has answered most of my needs until now.
2. It's well spread and has an active community.
3. I'm damn lazy when it comes to such mundane task.
Nonetheless Jim had made me curious and so I did give a look at the Redmine's features and I wasn't disappointed. It basically supports what Trac offers with some more interesting built-in features like Gantt chart, multiple projects, forums, DCVS, etc. Of course most of these features could be integrated to Trac easily thanks to the community (although I can't tell whether or not multiple projects in one Trac instance is feasible).
That being said not everything is perfect in this world and while discussing about this topic on the #kamaelia IRC channel, Matt Hammond, one of the Kamaelia long time project developer, linked me to a note from John Goerzen indicating that social considerations were sometimes as important as technical ones.
I guess you understand that I'm still struggling on which decision to make. Nevertheless redmine looks like a great product and if you're not using any software management tool yet I'm pretty sure you want to give a close look at it.
I wasn't very motivated by the idea of migrating from Trac to a different tool. Many reasons to that:
1. Trac has answered most of my needs until now.
2. It's well spread and has an active community.
3. I'm damn lazy when it comes to such mundane task.
Nonetheless Jim had made me curious and so I did give a look at the Redmine's features and I wasn't disappointed. It basically supports what Trac offers with some more interesting built-in features like Gantt chart, multiple projects, forums, DCVS, etc. Of course most of these features could be integrated to Trac easily thanks to the community (although I can't tell whether or not multiple projects in one Trac instance is feasible).
That being said not everything is perfect in this world and while discussing about this topic on the #kamaelia IRC channel, Matt Hammond, one of the Kamaelia long time project developer, linked me to a note from John Goerzen indicating that social considerations were sometimes as important as technical ones.
I guess you understand that I'm still struggling on which decision to make. Nevertheless redmine looks like a great product and if you're not using any software management tool yet I'm pretty sure you want to give a close look at it.
From subversion to mercurial
May 02, 2008
I've been wondering lately whether or not I should be moving my different projects from subversion to mercurial. Not so much because subversion could have let me down in any ways, I've been very happy with it, but rather because I wonder if it would help users of said projects to experiment and become occasional or regular contributors.
If anything, the distributed model lower the barrier for users to branch and go wild with ideas they might have. I don't really want to add contributors to my subversion repository just so that they can try something out that might not have any happy outcome. I've been happy enough that my projects had never drawn more attention that I could handle but that doesn't mean I don't want people to take over in some fashion and I've been wondering if DCVS like mercurial wouldn't allow that. Ultimately of course the official repository would be the one available on defuze but it would still be much easier for people to try out and propose modifications or extensions to projects.
Anyway, I would be happy in your feedback on the subject as I'm still trying to make my mind around this issue.
If anything, the distributed model lower the barrier for users to branch and go wild with ideas they might have. I don't really want to add contributors to my subversion repository just so that they can try something out that might not have any happy outcome. I've been happy enough that my projects had never drawn more attention that I could handle but that doesn't mean I don't want people to take over in some fashion and I've been wondering if DCVS like mercurial wouldn't allow that. Ultimately of course the official repository would be the one available on defuze but it would still be much easier for people to try out and propose modifications or extensions to projects.
Anyway, I would be happy in your feedback on the subject as I'm still trying to make my mind around this issue.
(Page 1 of 1, totaling 8 entries)
