deFuze.org

Via Peter Saint-André we learn that facebook is adding XMPP as a mean to connect and use its services. Independently from what usage is made of Facebook this is a great news for the Jabber community in general and the protocol itself as it demonstrates that, unlike some nay-sayers used to claim in the past, XMPP will be the the true sibling to HTTP when it comes to social networking and delivering near real-time data on a large scale. Indeed, considering that media companies like the BBC, are looking at using too (even if it's purely at a research level for now) it seems to me that XMPP has great years ahead.

I find that interesting though that Google, which has had a Jabber network for years now, has never been able to actually push much that way. I mean Orkut and Google Talk have been able to communicate for quite some time and yet it seems the Facebook prospect is more exciting for the Jabber community than Google's usage ever was. Maybe it comes down to the fact Orkut is a pale shadow of Facebook.

On a personal level, I won't complain that Google doesn't push advertisement through Google Talk of course but one may wonder why it doesn't do so from a business point of view. well probably they know having ads that way would drive their customers away, as they say (emphasis mine):

There are no ads in your chat sessions or your Quick Contacts list. Once a chat is saved, however, it becomes just like a Gmail message. And just as you may see relevant ads next to your Gmail messages, there now may be ads alongside your saved chats. Ads are only displayed when you're viewing a saved chat, and as with all ads in Gmail, they are matched entirely by computers. Only ads classified as Family-Safe are shown and we are constantly improving our technologies to prevent displaying any inappropriate ads. One of the things many Gmail users have told us is how much they appreciate the unobtrusive text ads in Gmail, as opposed to the large, irrelevant, blinking banner ads they often see in other services, and many have even cited the usefulness of the ads in Gmail.

We'll see how Facebook handles it considering the big fiasco Beacon was.

As a developer I've long felt frustrated at how limited the Google Talk standalone application is. Google Talk gadget is a bit better featured but it pains me having to use it when I prefer its big brother. Still both are very limited. To be fair though most well known IM clients speaking XMPP are limited in regards to what the protocol and its extensions offer. It's sad so little support PubSub and even those that do are somewhat basic in their support.

Let's hope we'll more and more XMPP applications out of the context of instant messaging or with a larger scope like microblogging.

Matthew Wood, from the BBC Radio Labs, posted a few days ago an exciting note regarding fun he was having with XMPP and services such as last.fm to inform user, via XMPP messages, of BBC radios broadcasting music they might like based on thier last.fm profile. I thought this was fantastic but I was even more excited when I read another note where he explained that he was using PubSub as a mean to carry and distribute metadata about BBC shows using Atom entries as the metadata format.

This evening I spent three hours expanding on the simple chat example coming with headstock to talk with the PubSub service. Then I integrated amplee as a way to offer an AtomPub interface at the same time. This means that when the demo starts both a XMPP client, connecting to Matt's service, and an AtomPub server, using amplee and served by CherryPy, are started. The XMPP client asks the server about PubSub nodes. For each node representing BBC channels I create an atompub collection within its own workspace. Simultaneously I subscribe to those nodes. I then ask the XMPP server for items belonging to those nodes and for each item, representing metadata about a show for instance, I create an atom entry that I store within the AtomPub store.

This means one can then simply subscribe with a feed reader to a given collection and/or a XMPP PubSub node. All of this happening on the fly starting from an empty AtomPub service document.

Eventually I will add support so that when an Atom entry is POSTed to a AtomPub collection, the according PubSub stanza is pushed towards the service (I doubt Matt's service accepts it though) allowing for microblogging support.

The source code of the example can be found here. If you want to understand how it works you might want to read this quick word I wrote about Kamaelia first which is at the core of headstock.

I'm pleased to announce the release of the first version of headstock, my XMPP implementation using the Kamaelia library. My main motivation behind working on that project rather than using one of the existing libraries was that I wanted to find a challenging project to work with Kamaelia. XMPP seemed challenging enough.

This first release is flagged as beta. Not production ready at all but offers enough so that you can play with it and actually use it for integrating XMPP into your applications. Documentation and tests are badly missing and they will eventually come but for now you'll have to go through the code. I know this is not attractive but I though it was worth to be released nonetheless. Enjoy.

A sad news for French speakers as the French O'Reilly editor has shutdown completely due to the lack of sales. They'll be missed.

Following my last post in regards to a move from subversion to mercurial I received many interesting comments. One key aspect that those comments showed was that no matter which DCVS I would use it would be critical to me that it could be interfaced with Trac since that's the tool I'm using to manage my projects. Jim Jones hinted that I could also see the problem the other way around and decide to change for a different software management tool that would be better at handling the DCVS I'd choose. He led me to discover Redmine.

I wasn't very motivated by the idea of migrating from Trac to a different tool. Many reasons to that:

1. Trac has answered most of my needs until now.
2. It's well spread and has an active community.
3. I'm damn lazy when it comes to such mundane task.

Nonetheless Jim had made me curious and so I did give a look at the Redmine's features and I wasn't disappointed. It basically supports what Trac offers with some more interesting built-in features like Gantt chart, multiple projects, forums, DCVS, etc. Of course most of these features could be integrated to Trac easily thanks to the community (although I can't tell whether or not multiple projects in one Trac instance is feasible).

That being said not everything is perfect in this world and while discussing about this topic on the #kamaelia IRC channel, Matt Hammond, one of the Kamaelia long time project developer, linked me to a note from John Goerzen indicating that social considerations were sometimes as important as technical ones.

I guess you understand that I'm still struggling on which decision to make. Nevertheless redmine looks like a great product and if you're not using any software management tool yet I'm pretty sure you want to give a close look at it.

I've been wondering lately whether or not I should be moving my different projects from subversion to mercurial. Not so much because subversion could have let me down in any ways, I've been very happy with it, but rather because I wonder if it would help users of said projects to experiment and become occasional or regular contributors.

If anything, the distributed model lower the barrier for users to branch and go wild with ideas they might have. I don't really want to add contributors to my subversion repository just so that they can try something out that might not have any happy outcome. I've been happy enough that my projects had never drawn more attention that I could handle but that doesn't mean I don't want people to take over in some fashion and I've been wondering if DCVS like mercurial wouldn't allow that. Ultimately of course the official repository would be the one available on defuze but it would still be much easier for people to try out and propose modifications or extensions to projects.

Anyway, I would be happy in your feedback on the subject as I'm still trying to make my mind around this issue.

At work we need to do some ldap work and it so happens we run Python 2.4. Unfortunately the only pre-compiled package I've found for Windows is for Python 2.5. I've tried all morning compiling python-ldap for Windows but that requires so much tweaking that I've gone slightly mad. Hence my request to the good Python community, would anyone have python-ldap for Python 2.4 on Windows (not cygwin by the way)?

Update: Well as an anonymous commenter showed me, the eggs for Python 2.4 linked from the python-ldap download page are indeed okay to work with. I had simply missed the DLLs package. I can only wish I could blame on being Friday.

There has been quite a lot of discussions around the use of XMPP in a web context and how to blend HTTP and XMPP protocols for a better social network experience.

I've been working on implementing XMPP for a while now on my spare time and even though it's been a much slower effort I would have liked, I've been able to have a fantastic fun with it. Most recently I've started integrating XMPP PubSub along with AtomPub using my AtomPub implementation, amplee.

The idea is to use amplee as an AtomPub library from within XMPP PubSub handlers. For instance I map AtomPub collections to PubSub nodes . When an atom entry is published to a node and that the client receives an acknowledgment of that publication, the handler uses amplee to store the entry in the AtomPub collection as well, respecting of course the RFC specification. Similarly if an item is deleted, the handler uses amplee to remove the entry from the collection. This works both ways, we can also run an AtomPub web service which issues the right XMPP stanzas according to the operation carried. So a POST on a collection would publish the atom entry to the XMPP server. The web service would obviously expose the Atom feed of the collection.

This is just a stub, but the idea here is to associate both protocols so that they cooperate to expand the audience of your network. It'd be easy to consider allowing people commenting to a blog entry using their XMPP client. You blog entry would for instance advert the XMPP service and node name where to publish items. The user could then subscribe to than node and publish items.

That's one reason why I had written amplee as an AtomPub library that could work outside of the HTTP protocol. RFC 5023 defines the protocol using HTTP but the root idea behind it works well in the context of XMPP and maybe AtomPub is the protocol that will rule them all.

Following my post on XMPP yesterday, one comment by Steven Kryskalla raised the point of the Mozilla extension: xmpp4moz. This extension is indeed an excellent showcase of what I wish could see more in the browser. However, while I knew about that extension I chose not to discuss about it as I wanted to stress over the need of a standard XMPP API, built-into browsers. This would imply a discussion shared by all the browser vendors and I thought inducing that Mozilla was already there could have disrupted the message. That being said xmpp4moz is a great piece of work and would definitely be an appropriate ground for more discussion.

What I think xmpp4moz does right is to use E4X to represent XMPP stanzas rather than providing its own datamodel. That means that the API stays quite at a low level but also means it's much quicker to grasp and more flexible.

I think ultimately what I'd like to see is a standard API that allows me from Javascript to do XMPP connection, authentication, session and binding. Then provide a nice interface to register handlers for given stanzas. By standard API I obviously meant that the same Javascript works the same way in any browser. Yeah I'm demanding like that.

Why Javascript? Two reasons that I can think of:

- The Ajax crazy party has pushed dramatically the competition across Javascript supporters and proved it was a valid and solid solution all along.
- Javascript is now the programing language with the best support inside browsers, no point going for something different.

Overall, considering how small XMPP-core is, I don't believe this is an impossible goal. The guys behind xmpp4moz have demonstrated it. What is tough with XMPP is not to provide support for the core but to start supporting its myriad of extensions that make the protocol so rich. But if people don't even have an access to the core they won't be able to build applications at all.

I've been a long fan of the XMPP protocol and I've started implementing it using the fantastic Kamaelia. With all the GSoC discussion around it appeared that lots of people were more and more interested in seeing XMPP becoming the natural partner of HTTP in the maze that the Internet has quickly become. Consequently Peter Saint-André created today the Social mailing-list to all people interested in discussing how XMPP could be used in what is that social web of yours.

I'm totally biased but I think there is more to XMPP than IM, the protocol and its suite of extensions provide great power applicable to RIA, whether they reside inside our outside the browser. For instance, I do believe that rather than using Comet one ought to use XMPP to push notifications to the client. In such a case one might consider the client as a node in a cloud of lazily interconnected nodes and then start thinking of the browser as more than a HTML rendering engine and without the resort to abuse HTTP for things it was never meant to support.

I wish browser vendors could start implementing XMPP within the browser as that would provide a fantastic incentive for more applications based on the power of XMPP.

Michael Schurter just posted a message on the main CherryPy users mailing-list asking developers using CherryPy to let the project team know about it. I want to support the idea as I would love our dusty success stories page being updated with new entries. I have a feeling that CherryPy is used quite a lot but mainly as a light HTTP framework in administration tools context and those projects quite likely are internals and don't have much visibility outside of their scope. Nonetheless we'd be interested in knowing where CherryPy is used so please let us know.

As per today's announcement. I'm glad to release amplee 0.6.0.

This release is an important move from previous releases as it doesn't include support for any HTTP layer out of the box anymore. The reason is that it made the previous API needlessly complex and stopped people to actually use amplee for what it aims at being: one simple representation of the AtomPub protocol server side. Basically I wish amplee was used as a library rather than as a host for AtomPub applications.

The 0.6.x branch will focus therefore on polishing the AtomPub model API as well as the related sub-packages such as the index and graph extension. Moreover I would like to improve the performance of amplee although they have already improved since 0.5.x. The graph sub-package is a first stab at using graph theory via the igraph package to perform foxy manipulations of Atom feeds.

One major change since 0.5.x is the move from bridge to Amara to parse, query and generate XML documents within amplee. I think that change was for the best considering the capabilities of Amara.

Another change is that I've dropped the INI file format for configuration and loading an amplee structure. Instead you can now directly use the XML service document itself and complete using a bit of extra code. That allows for some funny capabilities such as mirroring existing AtomPub service document (see the example directory for instance).

I would like to thank Eric Larson and Mohanaraj Gopala Krishnan for their feedback and patience. They have provided the project with a tremendous help.

== Download ==

easy_install -U amplee
Tarballs http://www.defuze.org/oss/amplee/
* svn co https://svn.defuze.org/oss/amplee/

== Documentation ==

Home page: http://trac.defuze.org/wiki/amplee
API: http://www.defuze.org/oss/amplee/api-0.6.0/
Tutorial (sort of): http://trac.defuze.org/wiki/amplee/Tutorial-0.6.x

Triggered by a question posted on the IronPython mailing-list by Jane Janet, I decided to see how IronPython would deal with OpenGL.

I quickly realized that the Tao framework was my best bet to gain access to OpenGL with .NET.

I therefore played a bit with the provided examples (ported from excellent tutorials such as NeHe) and started to port them to IronPython. I chose to use GLFW in order to get a context to run an OpenGL example rather than FreeGlut as I prefer the API design of GLFW.

I finally ported/wrote three examples on the IronPython cookbook as invited by Michael J. Foord.

The GLFW+OpenGL examples were easy to port and run.

However the SDL example has been quite a pain to setup. I think it comes down to the fact that Tao exposes the SDL API has an unmanaged code which means you have some conversion to do between IntPtr objects and SDL structures. Moreover I kept running into the SystemError: Missing or incorrect header for method GetValue exception when accessing structure attributes and that drove me nuts. Finally I settled for the horrible evt_type = Sdl.SDL_Event.type._get_(evt) call that basically says: in that structure, I want to grab the value of that field for the provided instance. Ugly I say.

Anyway I hope those few examples will help using IronPython in multimedia context without having to resort to Direct X.

I needed recently to grab an Atom feed of entris that contained link elements with a rel attribute set to "related", for some reason it's not that common and I therefore decided to make my own by scraping a few pages from Wikipedia, generating said link elements from the "See Also" sections of each articles. Here is a dirty script that performs such task.

# -*- coding: utf-8 -*-

from datetime import datetime

from urlparse import urljoin, urlparse

import uuid

from xml.sax.saxutils import escape



import amara

from BeautifulSoup import BeautifulSoup

import httplib2



visited_links = []



ATOM10_NS = u'http://www.w3.org/2005/Atom'

ATOM10_PREFIX = u'atom'



def qname(local_name, prefix=None):

    if not prefix:

        return local_name



    return u"%s:%s" % (prefix, local_name)

 



def init_feed():

   d = amara.create_document(prefixes={ATOM10_PREFIX: ATOM10_NS})

   feed = d.xml_create_element(qname(u"feed", ATOM10_PREFIX), ns=ATOM10_NS)

   d.xml_append(feed) 



   feed.xml_append(d.xml_create_element(qname(u"id", ATOM10_PREFIX), ns=ATOM10_NS,

                                        content=u'urn:uuid:' + unicode(uuid.uuid4())))



   feed.xml_append(d.xml_create_element(qname(u"updated", ATOM10_PREFIX), ns=ATOM10_NS,

                                        content=unicode(datetime.utcnow().isoformat())))

   return d, feed



d, feed = init_feed()



def run(url):

    print "Visiting: %s" % url

    

    entry = init_entry(url)

    h = httplib2.Http('.cache')

    r, c = h.request(url)

    soup = BeautifulSoup(c)



    entry.xml_append(d.xml_create_element(qname(u"content", ATOM10_PREFIX), ns=ATOM10_NS,

                                          attributes={u'type': u'text/html',

                                                      u'src': unicode(url)}))



    see_also = soup.find(name='a', attrs={'id': 'See_also'})

    if see_also:

        see_also_links = see_also.parent.findNextSibling(name='ul')

        if see_also_links: # some pages have the 
 empty 

            see_also_links = see_also_links.findAll('li')



        if see_also_links:

            next_to_visits = []

            for link in see_also_links:

                link = str(link.a['href'])

                if urlparse(link)[1] == '':

                    link = urljoin('http://en.wikipedia.org', link)

                    entry.xml_append(d.xml_create_element(qname(u"link", ATOM10_PREFIX), ns=ATOM10_NS,

                                                          attributes={u'rel': u'related', u'type': u'text/html',

                                                                      u'href': unicode(link)}))

                    if link not in visited_links:

                        visited_links.append(link)

                        next_to_visits.append(link)

                        

            for link in next_to_visits:

                run(link)



def init_entry(url):

    entry = d.xml_create_element(qname(u"entry", ATOM10_PREFIX), ns=ATOM10_NS)

    feed.xml_append(entry)

    

    entry.xml_append(d.xml_create_element(qname(u"id", ATOM10_PREFIX), ns=ATOM10_NS,

                                          content=u'urn:uuid:' + unicode(uuid.uuid5(uuid.NAMESPACE_URL, url))))

                 

    entry.xml_append(d.xml_create_element(qname(u"title", ATOM10_PREFIX), ns=ATOM10_NS,

                                          content=unicode(url), attributes={u'type': u'text'}))

   

    entry.xml_append(d.xml_create_element(qname(u"updated", ATOM10_PREFIX), ns=ATOM10_NS,

                                          content=unicode(datetime.utcnow().isoformat())))



    entry.xml_append(d.xml_create_element(qname(u"link", ATOM10_PREFIX), ns=ATOM10_NS,

                                          attributes={u'rel': u'self', u'type': u'text/html',

                                                      u'href': unicode(url)}))



    author = d.xml_create_element(qname(u"author", ATOM10_PREFIX), ns=ATOM10_NS)

    author.xml_append(d.xml_create_element(qname(u"uri", ATOM10_PREFIX), ns=ATOM10_NS,

                                           content=u'http://en.wikipedia.org'))

    entry.xml_append(author)



    return entry



if _name_ == '_main_':

    try:

        print "Ctrl-C to stop the scrapping"

        run('http://en.wikipedia.org/wiki/Castle')

    except KeyboardInterrupt:

        file('wikipedia_related_feed.atom', 'wb').write(feed.xml(indent=True))

        print "Saved as wikipedia_related_feed.atom"

    except Exception, ex:

        file('wikipedia_related_feed.atom', 'wb').write(feed.xml(indent=True))

        print "Saved as wikipedia_related_feed.atom"

        raise