Plugging AMQP and WebSockets

In my last article, I discussed the way the WSPBus could help your management of Python processes. This time, I’ll show how the bus can help plugging in heterogeneous frameworks and manage them properly too.

The following example will plug the WebSockets and AMQP together in order to channel data in and out of a WebSockets channel into AMQP exchanges and queues. For this, we’ll be using the Tornado web framework to handle the WebSockets side and pika for the AMQP one.

pika uses the Python built-in asyncore module to perform the non-blocking socket operations whilst Tornado comes with its own main loop on top of select or poll. Since Tornado doesn’t offer a single function call to iterate once, we’ll be directly using their main loop to block the process and therefore won’t be using the bus’ own block method.

Let’s see how the bus looks like

Next we create a plugin that will subscribe to the bus and which will be in charge for the AMQP communication.

The interesting bits are the amqp2ws and ws2amqp methods. The former is called anytime the AMQP broker pushes data to our AMQP consumer, we then use the bus to publish the message to any interested subscribers. The latter publishes to AMQP messages that come from the WebSockets channel.

Next let’s see the Tornado WebSockets handler.

The on_message method is called whenever data is received from the client, the push_message is used to push data to the client.

Finally, we setup the plug everything together:

Notice the fact we subscribe the asyncore poll function to the main channel of the bus so that pika works properly as if we had called asyncore.loop()

The code can be found here.

Managing your process with the CherryPy’s bus

CherryPy is a successful small web framework which over the years has built up its performances as well as its stability. To do so, Robert Brewer, the main CherryPy’s architect has introduced what is called the Web Site Process Bus (WSPBus). The idea is to manage a Python process by providing it with a bus to which one can publish or subscribe for events. CherryPy’s implementation of the bus comes with a set of pubsub handlers for very basic operations such as responding to system signals, handle thread creation and deletion, drop process privileges and handle PID files. The bus mechanism can help your handling of sub-processes so that they start, run and terminates gracefully. Let’s see how.

Create your bus

First, you need to create a bus instance. This could be as simple as this.

If you want to log through the bus, you will need further work since the bus doesn’t create a logger by default. Let’s see an example.

Not much, just creating a logger and subscribing the bus log channel to an instance method.

Associate the bus with the main process

Before we move on to the management of sub-process, let’s see how we can manage the main Python process already with our bus above.

For this, let’s imagine a bank placing stock orders, those orders will be handled by a broker running in a sub-process.

As you can see, not much again here, we simply associate a bus with the bank object. We also register to the exit channel of the bus so that when we terminated, we can do some cleanup. It’s good use to unregister from the bus.

We don’t actually care where those orders come from so we randomly generate them. The orders are placed every time the bus iterates its loop. This is done by attaching to the main channel of the bus.

We use a process queue to communicate with the broker’s sub-process.

Associate the bus with a sub-process

Handling the sub-process is actually similar to handling the main process. Let’s see the broker implementation for example.

Several things are to be noticed. First we register once again to the bus’ main channel a method that checks the shared queue for incoming data. Whenever the incoming message is “stop”, we exit the bus altogether, thus leaving the sub-process, since it was blocked on the bus loop.

Note that the stop method could be called by the parent process if you needed to programatically stop the sub-process.

Put it all together

Run the code above as follow:

This creates the shared queue, starts the sub-process that runs the broker and finally starts the bank within the main process.

You should see a bunch of messages in the console and if you hit Ctrl-C, this will stop both processes cleanly.

And here we are, we now manage processes and sub-processes with a clean solution. The CherryPy process bus is an elegant add-on to your toolbox that I can only highly advise to consider in the future. The WSPBus implementation is part of the main CherryPy package (CherryPy 3.x), so you’ll have to install it all, even if you don’t require the HTTP framework. But don’t let that hold you back since the HTTP framework isn’t required for the bus to be used.

Happy coding!

The code is here.

Is XMPP for the web ready?

A discussion started recently on the Jabber social mailing-list about the current state of XMPP support at Facebook. If you don’t remember the idea of a XMPP network for Facebook I’d say it’s rather normal considering they talked about it back in May 2008 and nothing has really happened since then. The discussion has quickly drifted toward the difficulty XMPP knows to really make it big and you can sense some despair within the community.

It seems there are several reasons for this.

First and foremost, XMPP is rather poorly integrated within browsers. Today you have to be part of the browser if you want to succeed. Perhaps the response will come from the rather brilliant lib Strophe. Strophe uses BOSH to connect to XMPP services since Javascript doesn’t offer much access to the low level socket connection object. I hope in the future browsers will offer a built-in XMPP API that can be accessed from Javascript allowing to avoid using BOSH.

Second of all, we need big players to start accepting to let go and start entering Jabber federations so that interoperability actually works. I assume companies have yet to find a way on how they can exercise control over the data they might expose through XMPP whilst finding a way to monetize them.

Third I’d say there is a critical lack of PR around XMPP as an added value to the business. To be honest there is the same issue with AtomPub in my opinion. Both are fantastic technologies but they don’t sell by themselves and I find there is a lack of support from companies to support them. SOAP might have been a crap technology but it was backed up by large businesses that were competing with each other (for the worse one might argue).

Finally, and to me this is the most critical reason, XMPP doesn’t integrate well with the web in general. From the browser side as I suggested above to the fact that it’s rather hard to combine web application with jabber ones. Its definitely doable but requires some careful attention to your architecture. Jack Moffitt argues it’s not XMPP’s fault. It’s quite true as a protocol but I believe it’s not really the right way to invite web developers to push it one step further if you blame them for doing it wrong.

I believe XMPP has tremendous potential but it’s still has some way to go before it finds its place.

XMPP and IronPython 2 using headstock, bridge and Kamaelia

I am glad to announce that IronPython 2 is now capable of running my XMPP
Python library: headstock.

.NET has already an excellent XMPP SDK called agsXMPP that is a native
.NET/C# framework. However I’m a Python developers at heart and I had
started quite a while ago writing my own XMPP library in Python using the
most excellent Kamaelia framework (designed for concurrency).

For a while IronPython had severe shortcomings that prevented it running
simple Kamaelia applications. Today I was able to run a simplechat demo
using a vanilla IP2 on Windows with only one single modification to the
logging module (thanks Seo). To be honest I didn’t expect it to go through
:)

The chat demo is simple enough but means more complex examples using XMPP
PubSub will work as well (they are all based on the same framework).

Now this isn’t production ready or anything. For instance the TLS support
is broken (hopefully something easy enough to fix) so you won’t be able to
connect to Google Talk for now.

Moreover I’m not sure the code is that fast considering how I had to
simulate an incremental XML parser atop System.Xml (this allows for a
XML stream to be parsed without requiring the full document or even
fragment to be read first).

This is a great news for me because it means I’ll be able to move ahead
with more work using IronPython 2.

Book Review – Expert Python Programming

Expert Python Programming by Tarek Ziadé is a great book that I recommend to anyone wishing to explore Python in a professional fashion.

The book covers topics that aren’t usual for Python books, at least to my knowledge, like how to build, package, release, distribute your software using tools that have become de facto standards in the Python community (setuptools, paste, zc.buildout, etc.). The book also gently introduces the reader to distributed version control and test-driven development.

The good

The book has 14 chapters and I really believe chapters 5 to 13 are worth every penny. They focus on the delivery of software from building to distributing through testing and optimizing. There is a little something for everyone there and I agree with the book saying it’s an authoritative reference on the subject. If you want an extensive resource on the process of moving from a personal project to a professional one, these chapters are what you need. Of course these chapters aren’t exhaustive but do offer enough meat to get you going and whet your appetite.

Tarek has a great experience in those fields an it transpires he surely wanted to include more content but that would have probably made the book harder to follow.

The “it depends on who you are and what you’ll be looking for”

I don’t want to say bad as it’s only my own opinion here but, I wasn’t convinced about chapters 1 to 4 nor was I about chapter 14. They deal with mainly advanced Python programming dos and donts: iterators, metaclasses, MRO, design patterns. Sadly, I didn’t feel they belonged to the topic of the book. They are interesting on their own but make the book less tight, as if it had started in one direction and suddenly shifted. That being said, they are useful as a reference and even if they aren’t always crystal clear, they are an enjoyable source of information.

A summary

Writing a book of this nature is hard because it’s large in scope but also in audience. Tarek explains that the book aims at Python developers but that in some instances project managers may find it useful. This is probably true if they were developers at some point. Expert Python Programming is a rich technical book which names isn’t too cocky considering its content.

Not only did I enjoy the book I also brought it to the office as it was immediatly useful to a case I was working on. Recommended.

Geolocalization and microblogging

I’ve been working recently with Adrian Hornsby who’s been interested in using the microblogging example I had setup to demonstrate headstock, amplee and some ideas about microblogging in general. Today Adrian asked me how to add some gelocalization information to a message flowing through the system. It took me just a couple of hours to implement it so that now you can push a message like: GEO text [lat,long] through your IM client. This will tell the demo to add a georss:point element to the generate atom entry which will eventually lead to a Google map to be displayed in the web page mapping the atom entry.

Notifixious – A notification platform

I’ve recently registered to Notifixious home page, a notification platform that integrates XMPP natively. Though the application is not looking as polished as one might hope it already provides interesting features.

Overall I think any work towards better notification system is the right idea. We need better ways to notify people or get notified about relevant piece of information. From a technology’s perspective transport protocols (well at the application’s level) already exist to perform the task of carrying the information, they are namely HTTP and XMPP. However there is still quite a large gap to fill about what to carry exactly. Notifixious doesn’t solve that gap per se but offers at least a good base for playing around with some floating ideas. For instance I’ve introduced today the Notifixious’s crowd to LLUP which has been initiated and carried on for a few years now to answer that specific problem. Perhaps it’ll make it way through.

Update: The guys at Notifixious just posted a message about how PubSub is heavily used by their service.

amplee 0.6.1

I’ve just released amplee 0.6.1.
This release is a minor release that fixes a few annoying defects and improves overall performances of the internal of amplee:

  • removes many of the usage of the copy.copy function (thanks to Mohanara Gopala Krishnan for his previous help)
  • drops the cmemcache module in favor the python-memcached one
  • adds a prune(member) method to the indexer API
  • extends with a few helpful functions the atompub API
  • adds more unit tests
  • drops the graph subpackage from the egg distribution (and I will remove it altogether for license issue in the next release)

If you’re using 0.6.0 I would advise to upgrade. The API is fully compatible and you’ll experience a more stable product.

I really wish to thank Mohan for his feedback, patience and will to send patches.

I will release a next version to clean out some remaining issue and than I will move toward amplee 0.7 that’ll support Amara 2.0 and will see some internal refactoring.

Maintenance and time zone

I have recently subscribed to the excellent Good Reads network and today I went to it only to see a message saying:

The system is down for maintenance as of 00:51 PDT.
It’ll be back shortly.

That’s okay, I mean it’s not such an essential website in my life that I have to get connected to it. I’ll visit it back. That being said I realised that if you plan on creating the next social network that everyone speaks about [1], make sure that you do your maintenance on a per-time zone basis. It’s cool to wait night in the USA to be the as little disruptive as you can but the rest of the World might not appreciate being put off in such fashion.

Of course scheduling maintenance based on a time zone means your infrastructure is designed to support it. Well, you said you wanted to be the next D.
[1] I don’t believe that’s something Good Reads is after mind you.