How to get on about it? Well that’s the surprise, CherryPy in itself doesn’t need to be patched. Granted I haven’t tried all the various tools provided by CherryPy but the server and the dispatching works just fine.

First, you need get the CherryPy source code, build and copy the resulting cherrypy package into the SL4A scripts directory.

Once you’ve plugged your phone to your machine through USB, run the next commands:

$ svn co http://svn.cherrypy.org/trunk cp3-trunk
$ cd cp3-trunk
$ python setup.py build
$ cp -r build/lib.linux-i686-2.6/cherrypy/ /media/usb0/sl4a/scripts/

Just change the path to match your environment. That’s it.

Now you can copy your own script, let’s assume you use something like below:

# -*- coding: utf-8 -*-
import logging
# The multiprocessing package isn't
# part of the ASE installation so
# we must disable multiprocessing logging
logging.logMultiprocessing = 0
 
import android
import cherrypy
 
class Root(object):
    def __init__(self):
        self.droid = android.Android()
 
    @cherrypy.expose
    def index(self):
        self.droid.vibrate()
        return "Hello from my phone"
 
    @cherrypy.expose
    def location(self):
        location = self.droid.getLastKnownLocation().result
        location = location.get('network', location.get('gps'))
        return "LAT: %s, LON: %s" % (location['latitude'],
                                     location['longitude'])
 
def run():
    cherrypy.config.update({'server.socket_host': '0.0.0.0'})
    cherrypy.quickstart(Root(), '/')
 
if __name__ == '__main__':
    run()

As you can see we must disable the multiprocessing logging since the multiprocessing package isn’t included with SL4A.

Save that script on your computer as cpdroid.py for example. Copy that file into the scripts directory of SL4A.

$ cp cpdroid.py /media/usb0/sl4a/scripts/

Unplug your phone and go to the SL4A application. Click on the cpdroid.py script, it should start fine. Then from your browser, go to http://phone_IP:8080/ and tada! You can also go to the /location path to get the geoloc of your phone.

The following recipe will try to make the example complete whilst as simple as possible to allow folks to start up with SQLAlchemy and CherryPy.

# -*- coding: utf-8 -*-
import os, os.path
 
import cherrypy
from cherrypy.process import wspbus, plugins
 
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column
from sqlalchemy.types import String, Integer
 
# Helper to map and register a Python class a db table
Base = declarative_base()
 
class Message(Base):
    __tablename__ = 'message'
    id = Column(Integer, primary_key=True)
    value =  Column(String)
 
    def __init__(self, message):
        Base.__init__(self)
        self.value = message
 
    def __str__(self):
        return self.value.encode('utf-8')
 
    def __unicode__(self):
        return self.value
 
    @staticmethod
    def list(session):
        return session.query(Message).all()
 
 
class SAEnginePlugin(plugins.SimplePlugin):
    def __init__(self, bus):
        """
        The plugin is registered to the CherryPy engine and therefore
        is part of the bus (the engine *is* a bus) registery.
 
        We use this plugin to create the SA engine. At the same time,
        when the plugin starts we create the tables into the database
        using the mapped class of the global metadata.
 
        Finally we create a new 'bind' channel that the SA tool
        will use to map a session to the SA engine at request time.
        """
        plugins.SimplePlugin.__init__(self, bus)
        self.sa_engine = None
        self.bus.subscribe("bind", self.bind)
 
    def start(self):
        db_path = os.path.abspath(os.path.join(os.curdir, 'my.db'))
        self.sa_engine = create_engine('sqlite:///%s' % db_path, echo=True)
        Base.metadata.create_all(self.sa_engine)
 
    def stop(self):
        if self.sa_engine:
            self.sa_engine.dispose()
            self.sa_engine = None
 
    def bind(self, session):
        session.configure(bind=self.sa_engine)
 
class SATool(cherrypy.Tool):
    def __init__(self):
        """
        The SA tool is responsible for associating a SA session
        to the SA engine and attaching it to the current request.
        Since we are running in a multithreaded application,
        we use the scoped_session that will create a session
        on a per thread basis so that you don't worry about
        concurrency on the session object itself.
 
        This tools binds a session to the engine each time
        a requests starts and commits/rollbacks whenever
        the request terminates.
        """
        cherrypy.Tool.__init__(self, 'on_start_resource',
                               self.bind_session,
                               priority=20)
 
        self.session = scoped_session(sessionmaker(autoflush=True,
                                                  autocommit=False))
 
    def _setup(self):
        cherrypy.Tool._setup(self)
        cherrypy.request.hooks.attach('on_end_resource',
                                      self.commit_transaction,
                                      priority=80)
 
    def bind_session(self):
        cherrypy.engine.publish('bind', self.session)
        cherrypy.request.db = self.session
 
    def commit_transaction(self):
        cherrypy.request.db = None
        try:
            self.session.commit()
        except:
            self.session.rollback()  
            raise
        finally:
            self.session.remove()
 
 
 
 
class Root(object):
    @cherrypy.expose
    def index(self):
        # print all the recorded messages so far
        msgs = [str(msg) for msg in Message.list(cherrypy.request.db)]
        cherrypy.response.headers['content-type'] = 'text/plain'
        return "Here are your list of messages: %s" % '\n'.join(msgs)
 
    @cherrypy.expose
    def record(self, msg):
        # go to /record?msg=hello world to record a "hello world" message
        m = Message(msg)
        cherrypy.request.db.add(m)
        cherrypy.response.headers['content-type'] = 'text/plain'
        return "Recorded: %s" % m
 
if __name__ == '__main__':
    SAEnginePlugin(cherrypy.engine).subscribe()
    cherrypy.tools.db = SATool()
    cherrypy.tree.mount(Root(), '/', {'/': {'tools.db.on': True}})
    cherrypy.engine.start()
    cherrypy.engine.block()

The general idea is to use the plugin mechanism to register functions on an engine basis and enable a tool that will provide an access to the SQLAlchemy session at request time.

The idea is to provide a simple Python API over the HBase one and couple it with a Python interpreter. This means, it offers the possibility to perform any Python (well Jython) operations whilst operating on HBase itself with an easier API than the Java one.

Note also that the tool uses the WSPBus already described in an earlier article to control the process itself. You will therefore need CherryPy’s latest revision.

# -*- coding: utf-8 -*-
import sys
import os
import code
import readline
import rlcompleter
 
from org.apache.hadoop.hbase import HBaseConfiguration, \
     HTableDescriptor, HColumnDescriptor
from org.apache.hadoop.hbase.client import HBaseAdmin, \
     HTable, Put, Get, Scan
 
import logging
from logging import handlers
 
from cherrypy.process import wspbus
from cherrypy.process import plugins
 
class StaveBus(wspbus.Bus):
    def __init__(self):
        wspbus.Bus.__init__(self)
        self.open_logger()
        self.subscribe("log", self._log)
 
        sig = plugins.SignalHandler(self)
        if sys.platform[:4] == 'java':
            del sig.handlers['SIGUSR1']
            sig.handlers['SIGUSR2'] = self.graceful
            self.log("SIGUSR1 cannot be set on the JVM platform. Using SIGUSR2 instead.")
 
            # See http://bugs.jython.org/issue1313
            sig.handlers['SIGINT'] = self._jython_handle_SIGINT
        sig.subscribe()
 
    def exit(self):
        wspbus.Bus.exit(self)
        self.close_logger()
 
    def open_logger(self, name=""):
        logger = logging.getLogger(name)
        logger.setLevel(logging.INFO)
        h = logging.StreamHandler(sys.stdout)
        h.setLevel(logging.INFO)
        h.setFormatter(logging.Formatter("[%(asctime)s] %(name)s - %(levelname)s - %(message)s"))
        logger.addHandler(h)
 
        self.logger = logger
 
    def close_logger(self):
        for handler in self.logger.handlers:
            handler.flush()
            handler.close()
 
    def _log(self, msg="", level=logging.INFO):
        self.logger.log(level, msg)
 
    def _jython_handle_SIGINT(self, signum=None, frame=None):
        # See http://bugs.jython.org/issue1313
        self.log('Keyboard Interrupt: shutting down bus')
        self.exit()
 
class HbaseConsolePlugin(plugins.SimplePlugin):
    def __init__(self, bus):
        plugins.SimplePlugin.__init__(self, bus)
        self.console = HbaseConsole()
 
    def start(self):
        self.console.setup()
        self.console.run()
 
class HbaseConsole(object):
    def __init__(self):
        # we provide this instance to the underlying interpreter
        # as the interface to operate on HBase
        self.namespace = {'c': HbaseCommand()}
 
    def setup(self):
        readline.set_completer(rlcompleter.Completer(self.namespace).complete)
        readline.parse_and_bind("tab:complete")
        import user
 
    def run(self):
        code.interact(local=self.namespace)
 
class HbaseCommand(object):
    def __init__(self, conf=None, admin=None):
        self.conf = conf
        if not conf:
            self.conf = HBaseConfiguration()
        self.admin = admin
        if not admin:
            self.admin = HBaseAdmin(self.conf)
 
    def table(self, name):
        return HTableCommand(name, self.conf, self.admin)
 
    def list_tables(self):
        return self.admin.listTables().tolist()
 
class HTableCommand(object):
    def __init__(self, name, conf, admin):
        self.conf = conf
        self.admin = admin
        self.name = name
        self._table = None
 
    def row(self, name):
        if not self._table:
            self._table = HTable(self.conf, self.name)
        return HRowCommand(self._table, name)
 
    def create(self, families=None):
        desc = HTableDescriptor(self.name)
        if families:
            for family in families:
                desc.addFamily(HColumnDescriptor(family))
        self.admin.createTable(desc)
        self._table = HTable(self.conf, self.name)
        return self._table
 
    def scan(self, start_row=None, end_row=None, filter=None):
        if not self._table:
            self._table = HTable(self.conf, self.name)
 
        sc = None
        if start_row and filter:
            sc = Scan(start_row, filter)
        elif start_row and end_row:
            sc = Scan(start_row, end_row)
        elif start_row:
            sc = Scan(start_row)
        else:
            sc = Scan()
        s = self._table.getScanner(sc)
        while True:
            r = s.next()
            if r is None:
                raise StopIteration()
 
            yield r
 
    def delete(self):
        self.disable()
        self.admin.deleteTable(self.name)
 
    def disable(self):
        self.admin.disableTable(self.name)
 
    def enable(self):
        self.admin.enableTable(self.name)
 
    def exists(self):
        return self.admin.tableExists(self.name)
 
    def list_families(self):
        desc = HTableDescriptor(self.name)
        return desc.getColumnFamilies()
 
class HRowCommand(object):
    def __init__(self, table, rowname):
        self.table = table
        self.rowname = rowname
 
    def put(self, family, column, value):
        p = Put(self.rowname)
        p.add(family, column, value)
        self.table.put(p)
 
    def get(self, family, column):
        r = self.table.get(Get(self.rowname))
        v = r.getValue(family, column)
        if v is not None:
            return v.tostring()
 
 
if __name__ == '__main__':
    bus = StaveBus()
    HbaseConsolePlugin(bus).subscribe()
    bus.start()
    bus.block()

To test the tool, you can simply grab the latest copy of HBase and run:

hbase-0.20.4$ ./bin/start-hbase.sh

Then you need to configure your classpath so that it includes all the HBase dependencies. To determine them:

$ ps auwx|grep java|grep org.apache.hadoop.hbase.master.HMaster|perl -pi -e "s/.*classpath //"

Copy the full list of jars and export CLASSPATH with it. (This is from the HBase wiki on Jython and HBase).

Next you have to add an extra jar to the classpath so that Jython supports readline:

$ export CLASSPATH=$CLASSPATH:$HOME/jython2.5.1/extlibs/libreadline-java-0.8.jar

Make sure you’ll install libreadline-java as well.

Now, that your environment is setup, save the code above under a script named stave.py and run it as follow:

$ jython stave.py
Python 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54) 
[Java HotSpot(TM) Server VM (Sun Microsystems Inc.)] on java1.6.0_20
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> c.table('myTable').create(families=['aFamily:'])
>>> c.table('myTable').list_families()
array(org.apache.hadoop.hbase.HColumnDescriptor)
>>> c.table('myTable').row('aRow').put('aFamily', 'aColumn', 'hello world!')
>>> c.table('myTable').row('aRow').get('aFamily', 'aColumn')
'hello world!'
>>> list(c.table('myTable').scan())
[keyvalues={aRow/aFamily:aColumn/1277645421824/Put/vlen=12}]

You can import any Python module available to your Jython environment as well of course.

I will probably extend this tool over time but in the meantime I hope you’ll find it a useful canvas to operate HBase.

First we create a web handler for Tornado that will return a web page containing the Javascript code that will connect and converse with our WebSockets endpoint following the WebSockets API.

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        username = "User%d" % random.randint(0, 100)
        self.write("""<html>
        <head>
          <script type='application/javascript' src='/static/jquery-1.4.2.min.js'> </script>
          <script type='application/javascript'>
            $(document).ready(function() {
              var ws = new WebSocket('ws://localdomain.dom:8888/ws');
              ws.onmessage = function (evt) {
                 $('#chat').val($('#chat').val() + evt.data + '\\n');                  
              };
              $('#chatform').submit(function() {
                 ws.send('%(username)s: ' + $('#message').val());
                 $('#message').val("");
                 return false;
              });
            });
          </script>
        </head>
        <body>
        <form action='/ws' id='chatform' method='post'>
          <textarea id='chat' cols='35' rows='10'></textarea>
          <br />
          <label for='message'>%(username)s: </label><input type='text' id='message' />
          <input type='submit' value='Send' />
          </form>
        </body>
        </html>
        """ % {'username': username})

Every time, the user enters a message and submits it too our WebSockets endpoint which, in return, will forward any messages back to the client. These will be appended to the textarea.

Internally, each client gets notified of any message through AMQP and the bus. Indeed the WebSockets handler are subscribed to a channel that will be notified every time the AMQP server pushes data to the consumer. A side effect of this is that the Javascript code above doesn’t update the textarea when it sends the message the user has entered, but when the server sends it back.

Let’s see how we had to change the Tornado application to support that handler as well as the serving of jQuery as a static resource (you need the jQuery toolkit in the same directory as the Python module).

 
if __name__ == '__main__':
    application = tornado.web.Application([
        (r"/", MainHandler),
        (r"/ws", WebSocket2AMQP),
        ], static_path=".", bus=bus)
 
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
 
    bus.subscribe("main", poll)
    WS2AMQPPlugin(bus).subscribe()
    bus.start()
    bus.block()

The code is here.

Once the server is running, open two browser windows and access http://localhost:8888/. You should be able to type messages in one and see them appears in both windows.

Note:

This has been tested against the latest Chrome release. You will need to either set the “localdomain.dom” or provide the IP address of your network interface in the Javascript above since Chrome doesn’t allow for localhost nor 127.0.0.1.

The following example will plug the WebSockets and AMQP together in order to channel data in and out of a WebSockets channel into AMQP exchanges and queues. For this, we’ll be using the Tornado web framework to handle the WebSockets side and pika for the AMQP one.

pika uses the Python built-in asyncore module to perform the non-blocking socket operations whilst Tornado comes with its own main loop on top of select or poll. Since Tornado doesn’t offer a single function call to iterate once, we’ll be directly using their main loop to block the process and therefore won’t be using the bus’ own block method.

Let’s see how the bus looks like

 class MyBus(wspbus.Bus):
    def __init__(self, name=""):
        wspbus.Bus.__init__(self)
        self.open_logger(name)
        self.subscribe("log", self._log)
 
        self.ioloop = tornado.ioloop.IOLoop.instance()
        self.ioloop.add_callback(self.call_main)
 
    def call_main(self):
        self.publish('main')
        time.sleep(0.1)
        self.ioloop.add_callback(self.call_main)
 
    def block(self):
        ioloop = tornado.ioloop.IOLoop.instance()
        try:
            ioloop.start()
        except KeyboardInterrupt:
            ioloop.stop()
            self.exit()
 
    def exit(self):
        wspbus.Bus.exit(self)
        self.close_logger()
 
    def open_logger(self, name=""):
        logger = logging.getLogger(name)
        logger.setLevel(logging.INFO)
        h = logging.StreamHandler(sys.stdout)
        h.setLevel(logging.INFO)
        h.setFormatter(logging.Formatter("[%(asctime)s] %(name)s - %(levelname)s - %(message)s"))
        logger.addHandler(h)
 
        self.logger = logger
 
    def close_logger(self):
        for handler in self.logger.handlers:
            handler.flush()
            handler.close()
 
    def _log(self, msg="", level=logging.INFO):
        self.logger.log(level, msg)

Next we create a plugin that will subscribe to the bus and which will be in charge for the AMQP communication.

class WS2AMQPPlugin(plugins.SimplePlugin):
    def __init__(self, bus):
        plugins.SimplePlugin.__init__(self, bus)
        self.conn = pika.AsyncoreConnection(pika.ConnectionParameters('localhost'))
        self.channel = self.conn.channel()
        self.channel.exchange_declare(exchange="X", type="direct", durable=False)
        self.channel.queue_declare(queue="Q", durable=False, exclusive=False)
        self.channel.queue_bind(queue="Q", exchange="X", routing_key="")
 
        self.channel.basic_consume(self.amqp2ws, queue="Q")
 
        self.bus.subscribe("ws2amqp", self.ws2amqp)
        self.bus.subscribe("stop", self.cleanup)
 
    def cleanup(self):
        self.bus.unsubscribe("ws2amqp", self.ws2amqp)
        self.bus.unsubscribe("stop", self.cleanup)
        self.channel.queue_delete(queue="Q")
        self.channel.exchange_delete(exchange="X")
        self.conn.close()
 
    def amqp2ws(self, ch, method, header, body):
        self.bus.publish("amqp2ws", body)
        ch.basic_ack(delivery_tag=method.delivery_tag)
 
    def ws2amqp(self, message):
        self.bus.log("Publishing to AMQP: %s" % message)
        self.channel.basic_publish(exchange="X", routing_key="", body=message)

The interesting bits are the amqp2ws and ws2amqp methods. The former is called anytime the AMQP broker pushes data to our AMQP consumer, we then use the bus to publish the message to any interested subscribers. The latter publishes to AMQP messages that come from the WebSockets channel.

Next let’s see the Tornado WebSockets handler.

class WebSocket2AMQP(websocket.WebSocketHandler):
    def __init__(self, *args, **kwargs):
        websocket.WebSocketHandler.__init__(self, *args, **kwargs)
        self.settings['bus'].subscribe("amqp2ws", self.push_message)
 
    def open(self):
        self.receive_message(self.on_message)
 
    def on_message(self, message):
        self.settings['bus'].publish("ws2amqp", message)
        self.write_message(message)
        self.receive_message(self.on_message)
 
    def on_connection_close(self):
        self.settings['bus'].unsubscribe("amqp2ws", self.push_message)
 
    def push_message(self, message):
        self.write_message(message)

The on_message method is called whenever data is received from the client, the push_message is used to push data to the client.

Finally, we setup the plug everything together:

if __name__ == '__main__':
    application = tornado.web.Application([
        (r"/ws", WebSocket2AMQP),
        ], bus=bus)
 
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
 
    bus.subscribe("main", poll)
    WS2AMQPPlugin(bus).subscribe()
    bus.start()
    bus.block()

Notice the fact we subscribe the asyncore poll function to the main channel of the bus so that pika works properly as if we had called asyncore.loop()

The code can be found here.

Create your bus

First, you need to create a bus instance. This could be as simple as this.

1
2

from cherrypy.process import wspbus
bus = wspbus.Bus()

If you want to log through the bus, you will need further work since the bus doesn’t create a logger by default. Let’s see an example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

import sys
import logging
from logging import handlers
 
from cherrypy.process import wspbus
 
class MyBus(wspbus.Bus):
    def __init__(self, name=""):
        wspbus.Bus.__init__(self)
        self.open_logger(name)
        self.subscribe("log", self._log)
 
    def exit(self):
        wspbus.Bus.exit(self)
        self.close_logger()
 
    def open_logger(self, name=""):
        logger = logging.getLogger(name)
        logger.setLevel(logging.INFO)
        h = logging.StreamHandler(sys.stdout)
        h.setLevel(logging.INFO)
        h.setFormatter(logging.Formatter("[%(asctime)s] %(name)s - %(levelname)s - %(message)s"))
        logger.addHandler(h)
 
        self.logger = logger
 
    def close_logger(self):
        for handler in self.logger.handlers:
            handler.flush()
            handler.close()
 
    def _log(self, msg="", level=logging.INFO):
        self.logger.log(level, msg)

Not much, just creating a logger and subscribing the bus log channel to an instance method.

Associate the bus with the main process

Before we move on to the management of sub-process, let’s see how we can manage the main Python process already with our bus above.

For this, let’s imagine a bank placing stock orders, those orders will be handled by a broker running in a sub-process.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

import random
import string
from multiprocessing import Process
 
class Bank(object):
    def __init__(self, queue):
        self.bus = MyBus(Bank.__name__)
        self.queue = queue
        self.bus.subscribe("main", self.randomly_place_order)
        self.bus.subscribe("exit", self.terminate)
 
    def randomly_place_order(self):
        order = random.sample(['BUY', 'SELL'], 1)[0]
        code = random.sample(string.ascii_uppercase, 4)
        amount = random.randint(0, 100)
 
        message = "%s %s %d" % (order, ''.join(code), amount)
 
        self.bus.log("Placing order: %s" % message)
 
        self.queue.put(message)
 
    def run(self):
        self.bus.start()
        self.bus.block(interval=0.01)
 
    def terminate(self):
        self.bus.unsubscribe("main", self.randomly_place_order)
        self.bus.unsubscribe("exit", self.terminate)

As you can see, not much again here, we simply associate a bus with the bank object. We also register to the exit channel of the bus so that when we terminated, we can do some cleanup. It’s good use to unregister from the bus.

We don’t actually care where those orders come from so we randomly generate them. The orders are placed every time the bus iterates its loop. This is done by attaching to the main channel of the bus.

We use a process queue to communicate with the broker’s sub-process.

Associate the bus with a sub-process

Handling the sub-process is actually similar to handling the main process. Let’s see the broker implementation for example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

from Queue import Empty
 
class Broker(Process):
    def __init__(self, queue):
        Process.__init__(self)
        self.queue = queue
        self.bus = MyBus(Broker.__name__)
        self.bus.subscribe("main", self.check)
 
    def check(self):
        try:
            message = self.queue.get_nowait()
        except Empty:
            return
 
        if message == "stop":
            self.bus.unsubscribe("main", self.check)
            self.bus.exit()
        elif message.startswith("BUY"):
            self.buy(*message.split(' ', 2)[1:])
        elif message.startswith("SELL"):
            self.sell(*message.split(' ', 2)[1:])
 
    def run(self):
        self.bus.start()
        self.bus.block(interval=0.01)
 
    def stop(self):
        self.queue.put("stop")
 
    def buy(self, code, amount):
        self.bus.log("BUY order placed for %s %s" % (amount, code))
 
    def sell(self, code, amount):
        self.bus.log("SELL order placed for %s %s" % (amount, code))

Several things are to be noticed. First we register once again to the bus’ main channel a method that checks the shared queue for incoming data. Whenever the incoming message is “stop”, we exit the bus altogether, thus leaving the sub-process, since it was blocked on the bus loop.

Note that the stop method could be called by the parent process if you needed to programatically stop the sub-process.

Put it all together

Run the code above as follow:

1
2
3
4
5
6
7
8
9

if __name__ == '__main__':
    from multiprocessing import Queue
    queue = Queue()
 
    broker = Broker(queue)
    broker.start()
 
    bank = Bank(queue)
    bank.run()

This creates the shared queue, starts the sub-process that runs the broker and finally starts the bank within the main process.

You should see a bunch of messages in the console and if you hit Ctrl-C, this will stop both processes cleanly.

And here we are, we now manage processes and sub-processes with a clean solution. The CherryPy process bus is an elegant add-on to your toolbox that I can only highly advise to consider in the future. The WSPBus implementation is part of the main CherryPy package (CherryPy 3.x), so you’ll have to install it all, even if you don’t require the HTTP framework. But don’t let that hold you back since the HTTP framework isn’t required for the bus to be used.

Happy coding!

The code is here.

But as it stands, superfeedr is just the tip of the iceberg. It could do much more for you.

Resource sharing

It would be probably be interesting if you could easily share with others (contacts, aggregators, social networks, etc.) links that you find interesting. This means you would become a source of information too. In other words, it’s as if you were generating a feed of links which others could subscribe too through superfeedr. But aside from the form itself, the possibility to automatically (re)publish interesting links is probably useful.

Recommendation support

Considering the sheer amount of feeds parsed by superfeedr and the fact they know the set of feeds you are interested in, it would be quite useful to add a recommendation mechanism in place. It means analysing feeds metadata but also giving weight to feeds based on the number of subscribers they have in the system or how long they retain subscribers. Similarly it would be relevant to publishers to get metrics to monitor how people consume their resources.

A dedicated client

XMPP clients suck. They all do when you want to go beyond pure instant messaging. PubSub is just barely supported and when it is, it’s just not user friendly. Therefore, following what the guys at seesmic have done, if you want your users to enjoy your service, you need a dedicated client. They care naught for your protocol. They want information displayed in the fashion du jour. These days, you gotta look like Twitter. Maybe Process One OneChannel could be a lead.

Integrate with Wave

That one is probably just for the fun of it but Wave is quite fancy-able these days and being able to be notified via a Wave could be a very cool.

Have a mobile presence

Mobiles will be where the market is in the next coming years. New generation mobile phones can support really well HTTP and even XMPP as BuddyMob has shown for quite some time with Android. Being able to broadcast to mobile phones is a must.

Joe argues that even though AtomPub hasn’t failed as technology, it hasn’t swept the inter-tubes world away as hoped probably because the rationale for creating such a protocol years back doesn’t really pan out any longer.

I share Joe’s slight disappointment at not seeing AtomPub more spread on the web but I don’t entirely agree with the reasons he puts forward:

So why hasn’t AtomPub stormed the world to become the one true protocol? Well, there are three answers:

• Browsers
• Browsers
• Browsers

The world is a different place then it was when Atom and AtomPub started back in 2002, browsers are much more powerful, Javascript compatibility is increasing among them, there are more libraries to smooth over the differences, and connectivity is on the rise.

I’m kind of puzzled, browsers did exist back in 2002, didn’t they? Sure they are more standardized, Javascript is not a shameful programing language like it used to be back in the DHTML days but nonetheless, browsers haven’t fundamentally changed since then. Moreover, through all the years required to release AtomPub nothing prevented the WG in charge of it to reach out for browser vendors. I don’t remember seeing much of them during the long discussions that took place. I find therefore rather harsh to complain at browsers for the fact AtomPub isn’t spread enough. (Note to the Hybi list, don’t make the same potential mistake).

Joe also suggests that XML was overcome by the couple: HTML+JSON. That is very true and Joe acknowledges that he didn’t think this would happen but when he says:

All of the advances in browsers and connectivity have conspired to keep AtomPub from reaching the widespread adoption that I had envisioned when work started on the protocol, but that doesn’t mean it’s a failure.

Conspired? Come on Joe. XMPP certainly suffers from the priority browser vendors have made in the past years but XMPP isn’t built atop HTTP when AtomPub is. I’m sorry Joe but, if anything, the fact Javascript has now so many fantastic libraries make it even simpler to interface with any protocol based on HTTP. Browsers have everything one needs to build an application conversing through AtomPub.

I disagree with Joe’s reasoning and personally I think there is a much bigger issue here to explain the protocol’s difficulties to be a premium choice for public web services.

Let’s review some social network platforms and the way they expose their data.

Wikipedia

Atom: Only used to view global recent changes and utter failure at actually using the format for what it’s good at. In a nutshell, everything is serialized to HTML and packed into the atom:summary element.

API: MediaWiki offers an API to expose and manipulate data which isn’t based on AtomPub but follows quite obviously the same design considering the document’s nature of a wiki. Note that the API supports several output formats but none of which is Atom.

Facebook

Atom: As far as I can tell, Facebook doesn’t offer any support for Atom.

API: Facebook has its own API which doesn’t really map well to AtomPub even though it’s quite often simple CRUD operations.

Twitter

Atom: Twitter exposes friend’s timeline as Atom feeds and does it well. Their API doesn’t support it however.

API: Twitter’s API is also not based on AtomPub.

Last.fm

Atom: No support for Atom.

API: Last.fm’s API doesn’t use AtomPub, well considering they don’t even use Atom itself, it kind of make sense.

Orkut

Atom: Available but not officially supported as per the documentation.

API: Orkut’s API is based on OpenSocial which if I’m not mistaken isn’t based on AtomPub.

MySpace

Atom: Not from  the public website as far as I could tell.

API: MySpace’s API is based on OpenSocial too, thus not based on AtomPub.

Flickr

Atom: Offers Atom feeds for a wide range of their data from their website.

API: Flickr has its own API not based on AtomPub and doesn’t output to Atom.

Google’s various API

Atom: Extensive support for it.

API: Extensively based on AtomPub even though no public support for the AtomPub service document which kind of limits service discovery.

I’m not trying to point fingers at those services, most of them came to life way before AtomPub became seriously defined (and maybe Tim Bray was right to warn the WG it was taken too long) so it’s not really a surprise it’s not used. In most cases though exposed APIs could be mapped to the protocol as well as the Atom format but obviously that would require major and fundamental architectural refactorings which make little economical sense.

It’s worth noting that some services have hard times even exposing data without degrading their values. Atom is a well-defined protocol that offers a simple view on web semantics. Used well, it offers a fantastic ground for mashup of organized and structured data. This is added value I’m wondering if mentioned social network platforms are willing to propose. In some respects Yahoo Pipes might not have to exist if Atom and AtomPub were properly used. Atom is so much more than pure serialization format. Atom can be serialized to JSON and still be Atom. Atom offers a real path to build powerful and meaningful social graph of distributed data.

Ultimately, data is money and I assume it’s a technical issue as much as an economical one. It’s easy to help building the piping for interoperability but it seems harder to actually open up the tap of data.

That being said, as Joe says:

There is still plenty of uses for AtomPub and it has quietly appeared in many places.

Atom and AtomPub aren’t dead but are probably less visible than one might have hoped, used in more traditional places and I suggest you follow Peter Keane to get a feel for it.

Ubikod is a young but enthusiastic company that is geared toward the future as they’re leading the way of social network for the Google Android platform through BuddyMob.

The team is passionate and has gathered quite a great knowledge of the platform. On top of that they’re using foxy technologies like XMPP for their platform and I’m excited about getting my hands on.

Things are looking great.

It seems there are several reasons for this.

First and foremost, XMPP is rather poorly integrated within browsers. Today you have to be part of the browser if you want to succeed. Perhaps the response will come from the rather brilliant lib Strophe. Strophe uses BOSH to connect to XMPP services since Javascript doesn’t offer much access to the low level socket connection object. I hope in the future browsers will offer a built-in XMPP API that can be accessed from Javascript allowing to avoid using BOSH.

Second of all, we need big players to start accepting to let go and start entering Jabber federations so that interoperability actually works. I assume companies have yet to find a way on how they can exercise control over the data they might expose through XMPP whilst finding a way to monetize them.

Third I’d say there is a critical lack of PR around XMPP as an added value to the business. To be honest there is the same issue with AtomPub in my opinion. Both are fantastic technologies but they don’t sell by themselves and I find there is a lack of support from companies to support them. SOAP might have been a crap technology but it was backed up by large businesses that were competing with each other (for the worse one might argue).

Finally, and to me this is the most critical reason, XMPP doesn’t integrate well with the web in general. From the browser side as I suggested above to the fact that it’s rather hard to combine web application with jabber ones. Its definitely doable but requires some careful attention to your architecture. Jack Moffitt argues it’s not XMPP’s fault. It’s quite true as a protocol but I believe it’s not really the right way to invite web developers to push it one step further if you blame them for doing it wrong.

I believe XMPP has tremendous potential but it’s still has some way to go before it finds its place.