Robot Framework and Sphinx: A suitable toolset for your specification by example

At work, we have been using Robot Framework for all kinds of tests for a few years now and it’s proven to be the good choice. Robot Framework’s simple syntax and grammar does not scare testers away (usually). At the same time, its design makes it easy to support complex use cases as well as simple ones through the power of the Python programming language.

One blind spot however, in my opinion anyway, is the way Robot Framework let you document your tests. It provides a section for this, with basic HTML support but it has always felt limited and not really friendly.

Luckily, in the recent releases, the Robot Framework developers have provided a built-in support for reStructuredText. Not that the documentation section supports this syntax, but instead, you can embed Robot Framework tests into a reStructuredText document, and therefore into Sphinx as well.

The gain isn’t so much visible in the Robot Framework reports since the reStructuredText sections won’t appear in those, but it means you can generate HTML documents which embed executable tests. Fans of doctests will be in known territory.

I think this is a powerful combination as it bridges the tests with the specifications and ensure they are both kept locally at the same place, imrpoving their chance to stay synchronised.  In my mind, it provides a great framework to follow the Specification by Example that Gojko Adzic described so eloquently.

Here is a simple:

Finally, a related powerful extension provides a simple mechanism to include Robot Framework tests into Sphinx documentation. We use it extensively at work as we wanted to keep our tests outside in distinct files without losing the ability to see them embedded into the generated HTML documentation.

CherryPy documentation new start

Early on this year, a discussion emerged on the CherryPy mailing-list about the project. Most people said they loved the project but had struggled with its documentation. Though rich and extensive, it was felt it left down the project somehow by being not designed in a way that was attractive to new comers. I took upon myself to rewrite it from scratch following some ideas exchanged on the mailing-list.

The general expressed wish was to make it friendliers to people starting with the framework whilst making easy to look for common tasks and patterns. This suited me well as I wanted to carry the work I started on the various recipes I keep on BitBucket.

Eventually, I quickly wrote a set of tutorials to guide people through the general layout of a CherryPy application. Then I developed upon the recipes idea by going through many of the most recurrent questions we have on the mailing-list. Finally, I wrote an extensive section regarding the core features of the framework: plugins, tools, the bus, the dispatchers, etc. Those features are seldom used to their best even though they provide a very powerful backbone to design your application in clean way.

The documentation is now online and seems to have been well-received. It will need to be completed but I believe they already make the project much more appealing and fun to work with.

Having fun with WebSocket and Canvas

Recently, I was advised that WebFaction had added support for WebSocket in their custom applications by enabling the according nginx module in their frontend. Obviously, I had to try it out with my own websocket library: ws4py.

Setting up your WebFaction application

Well, as usual with WebFaction, setting up the application is dead simple. Only a few clicks from their control panel.

Create a Custom application and select websocket. This will provide you with a port that your backend will be bound to. And voilà.

Now, your application is created but you won’t yet be able to connect a websocket client. Indeed, you must associate a domain or subdomain with that application.

It is likely your application will be used from a javascript connector in living in a browser, which means, you will be bound by the browser same-origin security model. I would therefore advise you to carefully consider your sub-domain and URL strategies. Probably something along:

  • http://yourhost/ : for the webapp
  • http://yourhost/ws : as the base url for all websocket endpoints

This is just a suggestion of course but this will make it easier for your deployment to follow a simple strategy like this one.

In the WebFaction control panel, create a website which associates your web application with the domain (your webapp can be anything you need to). Associate then your custom websocket application with the same domain but a different path segment. Again, by sharing the same domain, you’ll avoid troubles regarding working around the same-origin security model. I would probably advise as well that you enable SSL but it’s up to you to make that decision.

Once done, you will have now a configured endpoint for your websocket application.

The Custome websocket application will forward all requests to you so that you can run your web and websocket apps from that single port. This is what the demo below allows itself doing. I would recommend that you run two different application processes, one for your webapp and another one for your websocket endpoint. Be antifragile.

Drawing stuff collaboratively

I setup a small demo (sorry self-signed certificate) to demonstrate how you can use HTML5 canvas and websocket features to perform collaborative tasks across various connected clients.

That demo runs a small webapp that also enables a websocket endpoint. When you are on the drawing board, everything you draw is sent to other connected clients so that their view reflects what’s happening on yours. Obviously this goes in any way frm any client to any other clients.

The demo is implemented using ws4py hosted within a CherryPy application. Drawing events are serialized into a json structure and sent over to the server which dispatches them to all participants of that board, and only that board (note, this demo doesn’t validate the content before dispatching back so please conservative with whom you share your board with).

Open the link found in the demo page and share it on as many browsers as you can (including your mobile device). Starting drawing from one device will make all other devices been drawn onto simultaneously and synchronously. Note that the board stays available for up to 5mn only and will disconnect all participants then.

The source code for the demo is located here.

Some feedback…

Let me share a bit of feedback about the whole process.

  • WebSockets are finally a reality. Unless you’re running an old browser or old mobile platform, RFC6455 is available to you.  This means, you can really leverage the power of push from the server. Mind you, you might want to look at Server-Side Event  as well.
  • There isn’t yet a clear understanding on how to properly configure your server resources. In my demo, the whole webapp also hosts the websocket app but this is probably not a good idea if you have a large amount of connected clients or intensive work done server side. Even though the initial connection is initiated from a HTTP request, I would suggest the websocket server is disconnected from the HTTP server process.
  • Security wise, I would suggest you follow the usual principles of validating that any data coming through before you process or dispatch them back.
  • WebFaction supports for websocket is dead easy to setup and fast (at least, since my demo is hosted in Europe and I live in France, I almost cannot see any delay). I would consider their performances good enough to support some really funky real-time applications.
  • jCanvas is really useful to unify your canvas operations.  For this demo, it’s been a blessing.
  • Device motion events are low level and you need to do a lot of leg work to actually make sense of them. This demo is probably not making a really good use of them.
  • There seems to be no universal way to detect that you are running on a mobile device. Go figure.

Next, I wouldn’t mind adding websocket to that fun demo from the mozilla developer network.

The joy of distributing Python packages for Python 2 and 3

I released ws4py 0.3.4 this weekend and although I had integrated support for Python 2 and 3 for a long time now, I ran into a challenge I had quite missed. Indeed, until now my Python 3 support had mainly been concerned about string handling and various compatibility modules. This has proven to work very well and avoided having to rely on external packages such as six.

However, in the past few weeks and I added asyncio support to ws4py and therefore introduced the newly yield from statement. Of course, this isn’t tolerated by Python 2 which complains with a well deserved SyntaxError.

The issues however is that I wished to distribute the same source code with a single source distribution archive. Initially, I had written a function that was preventing modules using that statement to actually be packaged. However, this was rather daft since that, if it weren’t packaged, it wouldn’t be distributed either. Next, I decided to some of setuptools magic. Well, it didn’t help since that’s not what it’s there for anyhow. At this stage, I should say: Don’t simply copy/paste. It will do no good.

Finally, I opted for a fairly simple solution. I knew that when a package is installed from a distribution packages, it is obviously built first. I therefore had to act after Python modules had been gathered but before they would be built. After briefly browsing through distutils source code, I found where I would perform surgery: the find_package_modules of the distutils.command.build_py.build_py class. The nice aspect of this solution is that the source distribution contains indeed all the modules, whether they aim Python 2 or 3 but it’s only when installed that the appropriate modules will be selected and built.

Am I doing it wrong? Is there a cleaner, nicer more pythonic way? If so, please let me know. If not, I hope this may help others that want a simple solution to handle their Python 2 and 3 modules in a single baseline.

“Robot Framework Test Automation” book review

From time to time PacktPub will request a book review of one of their Python-related titles. This time around it was regarding their “Robot Framework Test Automation” book they recently released. Since I’ve been using this awesome acceptance testing tool at work for more than two years, I was happy to comply.

In a nutshell, Robot Framework provides a great interface that acts as the middle-man between variour stakeholders. Indeed, tests are written in plain text (though other formats are supported, I never use them) with a rather minimal set of rules making it (almost) straightforward to read even by non-technical persons. The dirty technical details being hidden away and implemented in Python and executable in one of the various Python VM (CPython, Jython, IronPython are supported out of the box).

Most of the time, the basics of the Robot Framework data model and workflow can be taught in a couple of hours. However, being efficient with it will take a little more time. Still, people don’t have to learn a complete programming language (Python) itself and that’s a relief meaning they are happy to work with Robot Framework sometimes cumbersome syntax.

In spite of having a rather extensive documentation available online, the project did lack a good, straight to the point summary that takes you by the hand. Moreover, the documentation’s style of the project is fairly dry and Unix-style making it tedious to browse sometimes. Still, the content is there and it rarely failed me. With that said, having a friendly book on the subject is a great thing. Kudos to PacktPub. Now about the book…

The good

The book provides an introduction to the tool, its most common usages and even tries to guide you getting more from it. It’s a short book, 83 pages, that will not bore you with complex details. In other words, it’s a good companion of the online documentation if you start with Robot Framework.

Sumit Bisht, the author, does a good job keeping a neutral point of view in regards to how you should use Robot Framework. Indeed, depending on your software under test, you might want to have a more data-oriented approach (ala fitness), a behavior-driven testing approach or even a more assert-oriented style. Not many software can deal with all of them equally and it depends also on how testing is perceived in your organisation. Robot Framework can cope with all of them.

The bad

Though I could understand it’s only an introduction, it feels like some concepts are not properly explored. The idea behind keywords, the internal data model, dynamic libraries, etc. In other words, you will not really understand the underlying blocks and axioms that are the pedestal of the whole tool, you’ll rather learn the basics of using it. In fact, the only section where the book goes into more technical details (with a good example on using sikuli) will probably confuse you since it failed to properly introduce the principles behind them.

The ugly

There isn’t anything particulary that bad with this book, again it should be considered as a friendly introduction. I do not agree with a few minor points Sumit makes but they hardly matter and aren’t wrong anyway, just a matter of opinion. Note also that the book lacks examples a couple of times where it would have mattered but I don’t believe this makes the book any less useful.

The only thing that annoys me really is that PacktPub book’s layout still looks so unprofesionnal. They should really make an effort as the code is, most of the time, too hard to read (actually on this one item, it wasn’t that bad).

Final note

I think this book is ideal if you are about to start with Robot Framework as it will speed up the basics. If you’re already used to the tool, I am not sure it will help very much.




ws4py – WebSocket client and server library for Python

Recently I released ws4py, a package that provides client and server WebSocket support for Python 2.6 and 2.7.

Let’s first have a quick overview of what ws4py offers for now:

  • WebSocket specification draft-10 of the current specification.
  • A threaded client. This gives a simple client that doesn’t require an external dependency.
  • A Tornado client. This client is based on Tornado 2.0 which is quite a popular way of running asynchronous networking code these days. Tornado provides its own server implementation so I didn’t include mine in ws4py.
  • A CherryPy extension so that you can integrate WebSocket from within your CherryPy 3.2.1 server.
  • A gevent server based on the popular gevent library. This is courtesy of Jeff Lindsay.
  • Based on Jeff’s work, a pure WSGI middleware as well (available in the current master branch only until the next release).
  • ws4py runs on Android devices thanks to the SL4A package

Hopefully more client and servers will be added along the way as well as Python 3.x support. The former should be rather simple to add due to the way I designed ws4py.

The main idea is to make a distinction between the bytes provider and the bytes processing. The former is essentially reading and writing bytes from the connected socket. The latter is the function of making something out of the received bytes based on the WebSocket specification. In most implementations I have seen so far, both are rather heavily intertwined making it difficult to use a different bytes provider.

ws4py tries a different path by relying on a great feature of Python: the possibility to send data back to a generator. For instance, the frame parsing yields the quantity of bytes each time it needs more and the caller feeds back the generator those bytes once they are received. In fact, the caller of a frame parser is a stream object which acts the same way. The caller of that stream object is in fact the bytes provider (a client or a server). The stream is in charge of aggregating frames into a WebSocket message. Thanks to that design, both the frame and stream objects are totally unaware of the bytes provider and can be easily adapted in various contexts (gevent, tornado, CherryPy, etc.).

On my TODO list for ws4py:

  • Upgrade to a more recent version of the specification
  • Python 3.x implementation
  • Better documentation, read, write documentation.
  • Better performances on very large WebSocket messages

Acceptance testing a CherryPy application with Robot Framework

I recently received the Python Testing Cookbook authored by Greg L. Turnquist and was happy to read about recipes on acceptance testing using Robot Framework. We’ve been using this tool at work for a few weeks now with great results. Greg shows how to test a web application using the Selenium Library extension for Robot Framework and I thought it’d be fun to demonstrate how to test a CherryPy application following his recipe. So here we go.

First some requirements:

Let’s define a simple CherryPy application, which displays a input text where to type a message. When the submit button is pressed, the message is sent to the server and returned as-is. Well it’s an echo message really.

Save the code above in a module named

Next, we create an extension to Robot Framework that will manage CherryPy. Save the following in a module It’s important to respect that name since Robot Framework expects the module and its class to match in names.

Note that we start and stop the CherryPy server during the test itself, meaning you don’t need to start it separately. Pure awesomeness.

Finally let’s write a straightforward acceptance test to validate the overall workflow of echoing a message using our little application.

Save the test above into a file named testmyapp.txt. You can finally run the test as follow:

This will start CherryPy, Selenium’s proxy server and Firefox within which the test case will be run. Easy, elegant and powerful.

Hosting a Django application on a CherryPy server

Recently at work I’ve had the requirement to host a Django application in a CherryPy server. I first looked for various projects I knew were doing just that. Unfortunately, after trying them I was rather disapointed. Their approach is to provide a command similar to the famous Django runserver‘s one but I’ve found it to be more complex than necessary. So I wrote my own module that performs those operations by staying much closer to how CherryPy does work, most specifically by using the process bus coming with CherryPy.

I’m sharing a stripped down version of the module I wrote which shows how one could host a Django application in a CherryPy server. Hopefully this might help some of you.

You can find the code along side a minimal Django application showing how this works here (BSD licence). I used Django 1.3 to generate a default project but the code above works well with older version of Django.

Edit 16/03/2012: Thanks to Damien Tougas, I’ve wrapped up a better recipe for hosting a Django application into a CherryPy application server.

Should we assess OpenData innovation and impacts?

Yesterday, I was at a talk titled: “OpenData, basis for a new political technologies ?” at la Cantine Numérique Rennaise [fr], a place about the digital age located in Rennes. During the debate, I asked how we could assess the impacts of OpenData without some sort of measuring instruments. This is question the EU asked itself in a recent report.

Xavier Crouan, who has been digital innovative and Informations director in Rennes for the past few years and has communicated extensively on OpenData, made a comment that I felt was a misunderstanding of my own question. He roughly stated that it felt typically French to request for tools, indicators whenever risks and innovation were taken. He believed this was saddening to hear French engineers being so grounded and felt innovation should not have to justify itself.

Honestly that wasn’t what I was leading at. The discussion at that point of the debate was about how OpenData would eventually make a difference in people’s life politically as much as economically. In that context, it seemed sensible to ask how we could measure the impacts of OpenData so that we could tweak, tune, improve its usage.

Now in regards to innovation itself, I believe you usually need simple indicators to gauge whether or not you’re walking onto a fruitful path.

For instance, Rennes has held a contest for building applications on data it has recently opened. Xavier Crouan has indicated that 2000 people had voted. One might consider it is an indicator whether or not the contest was publicly a success and if not, how to tune it if there’s another contest next year.

Shooting in different directions in hope one path will lead to strong innovation is shortsighted in my book. You need to define a few criteria that will assess how each direction fares. This is what OpenData promotes too: improving efficiency in reusing of public sector data.

Innovation is not incompatible with retrospective.

Thoughts on technical strategies for an OpenData world

I’ve been part of a very interesting discussion on a French forum around some technical aspects of the opening of public data. To provide a bit of context, the French city of Rennes, Brittany, has been leading the way in France in 2010 on the subject of opening its data [fr]. The city has released local authorities and transport datasets. The former are cold datasets served as structured files while the latter is a mix of hot and cold datasets provided through a dedicated API [fr].

The whole discussion started from that API which is basically a RPC web service that returns data such as:

  • Current number of available bikes at a public bike station
  • Current number of free slots to park a bike at a station
  • Bus and subway transit status
  • Location of travel card/ticket vendors

Someone on the forum [fr] wondered why the API wasn’t following a more RESTful approach to which I chimed in backing up that idea. Then the discussion went off whether or not REST would bring any added value in comparison to the RPC web service in place. That got me thinking about one side of OpenData that is not too often put ahead, naming the “how”. Indeed, mostly, discussions around OpenData deal with the kind of data to release, the reason why and under which conditions.

However it should not be forgotten that releasing public data isn’t a trivial operation, even when local authorities and other public bodies do have datasets at hand.

Data source constraints

By talking to folks working in public agencies you quickly realise that they meet many issues even when they are determined to open up their datasets.

Old information system or none at all

It’s not uncommon that local authorities don’t have a proper information system. They may deal with documents such as Excel files or their system is just not designed to handled some modern operations.

Information System Lock-in

In some cases, namely with GIS, local authorities don’t even really have a way to perform export operations. When that happens it makes the release process more complicated and delay it.

Unfit data model

We often cry for more data to be opened but we fail to understand that the model to publicly expose data isn’t necessarily the one used internally by local authorities.

Data provider strategies

One critical question local authorities must ask themselves is whether or not they want to engage into the data/web-service game. In other words, do they want to give an access, as raw as possible to their data and be left alone, or do they want to have control and perhaps create something richer. This is not a light question as the implications can seriously impact the level of commitment a local authority will have to show.

Flat data provider

In that configuration the local authority decides to provide the lowest level of features by putting files (flat or structured) on the web for anyone to download. The advantages is the simplicity of deployment and maintenance as well as a very low cost. However this might also create a barrier for non technical folks who would want to provide innovative services as they would be required to process those files first.

This is how Rennes has done for its GIS datasets and, indeed, everyone I met had to perform operations on those files before they could start on their services.

Raw data web service

By raw I mean a web service that stays rather close of the dataset it serves. The advantage is the rather simplicity to setup such a service and the low cost it requires. In some cases such a web service might be just enough if the data model exposed by the web service doesn’t force the developer to create his own intermediary datamodel since, in that case, the web service isn’t better than flat files.

The city of Rennes uses that kind of service for some of its transit data.

Rich web service

Here the provider decides to go the extra mile by offering richer web service. In that case, the local authority defines a much richer data model on top of its own internal data structures. The big advantage is that some common functionalities are provided by the service and allows the developer to build much richer applications too. For instance rather than solely providing a GTFS file for its public transportation a developer must download, parse and create his own model, a provider might offer an API to query that dataset for useful information such as “what is the fastest way to go from A to B whilst avoiding traffic and with the lowest impact on the environment?”.

The city of Rennes doesn’t provide a rich web service  which can be explained by the fact that defining a rich model requires you have a good idea of how the data may be exploited. If you take some of Google’s APIs, it has taken some time before they reached their current matureness. Moreover Rennes follows the common pattern of “release often, release early” and the current API was released so that developers could rapidly start playing with it.

Data support

This is an aspect that is almost never approached. Should a local authority be accountable for data integrity, updates, support? The question is rather complex since it begs the question whether or not it’s within the local authority’s role to support those functions.

Implicitly this also leads to question whether or not a local authority should become a full fledged data provider. Should we start creating dedicated public, or even private, bodies to perform those tasks?

Which strategy to follow?

Eventually the decision is made based on various factors such as the kind of data a local authority owns, how it’s stored, its accessibility but also a question of cost, time and politics. I strongly believe that not thinking enough about how it should be distributed will impair the way the OpenData movement is perceived on the long run. Indeed, let’s not forget that, at least in France, the biggest consumer of public data are public services themselves and to win the OpenData uptake, the “how” must be carefully designed and implemented.

If public data are too complex to consume and process, OpenData will partly fail I believe.