Acceptance testing a CherryPy application with Robot Framework

I recently received the Python Testing Cookbook authored by Greg L. Turnquist and was happy to read about recipes on acceptance testing using Robot Framework. We’ve been using this tool at work for a few weeks now with great results. Greg shows how to test a web application using the Selenium Library extension for Robot Framework and I thought it’d be fun to demonstrate how to test a CherryPy application following his recipe. So here we go.

First some requirements:

$ mkvirtualenv --distribute --no-site-packages --unzip-setuptools acceptance
(acceptance)$ pip install cherrypy
(acceptance)$ pip install robotframework
(acceptance)$ pip install robotframework-seleniumlibrary

Let’s define a simple CherryPy application, which displays a input text where to type a message. When the submit button is pressed, the message is sent to the server and returned as-is. Well it’s an echo message really.

import cherrypy
 
__all__ = ['Echo']
 
class Echo(object):
    @cherrypy.expose
    def index(self):
        return """<html>
<head><title>Robot Framework Test for CherryPy</title></head>
<body>
<form method="post" action="/echo">
<input type="text" name="message" />
<input type="submit" />
</form>
</body>
</html>"""
 
    @cherrypy.expose
    def echo(self, message):
        return message
 
if __name__ == '__main__':
    cherrypy.quickstart(Echo())

Save the code above in a module named myapp.py

Next, we create an extension to Robot Framework that will manage CherryPy. Save the following in a module CherryPyLib.py. It’s important to respect that name since Robot Framework expects the module and its class to match in names.

import imp
import os, os.path
 
import cherrypy
 
class CherryPyLib(object):
    def setup_cherrypy(self, conf_file=None):
        """
        Configures the CherryPy engine and server using
        the built-in 'embedded' environment mode.
 
        If provided, `conf_file` is a path to a CherryPy
        configuration file used in addition.
        """
        cherrypy.config.update({"environment": "embedded"})
        if conf_file:
            cherrypy.config.update(conf_file)            
 
    def start_cherrypy(self):
        """
        Starts a CherryPy engine.
        """
        cherrypy.engine.start()
 
    def exit_cherrypy(self):
        """
        Terminates a CherryPy engine.
        """
        cherrypy.engine.exit()
 
    def mount_application(self, appmod, appcls, directory=None):
        """
        Mounts an application to be tested. `appmod` is the name
        of a Python module containing `appcls`. The module is
        looked for in the given directory. If not provided, we use
        the current one instead.
        """
        directory = directory or os.getcwd()
        file, filename, description = imp.find_module(appmod, [directory])
        mod = imp.load_module(appmod, file, filename, description)
        if hasattr(mod, appcls):
            cls = getattr(mod, appcls)
            app = cls()
            cherrypy.tree.mount(app)
        else:
            raise ImportError, "cannot import name %s from %s" % (appcls, appmod)

Note that we start and stop the CherryPy server during the test itself, meaning you don’t need to start it separately. Pure awesomeness.

Finally let’s write a straightforward acceptance test to validate the overall workflow of echoing a message using our little application.

***Settings***
Library	SeleniumLibrary
Library	CherryPyLib
Suite Setup	Start Dependencies
Suite Teardown	Shutdown Dependencies
Test Setup	Mount Application	myapp	Echo

***Variables***
${MSG}	Hello World
${HOST}	http://localhost:8080/

***Test Cases***
Echo ${MSG}
     Open Browser	${HOST}
     Input text		message		${MSG}
     Submit form
     Page Should Contain		${MSG}
     Close All Browsers

***Keywords***
Start Dependencies
    Setup Cherrypy
    Start CherryPy
    Start Selenium Server
    Sleep 	3s

Shutdown Dependencies
    Stop Selenium Server
    Exit CherryPy

Save the test above into a file named testmyapp.txt. You can finally run the test as follow:

(acceptance)$ pybot --pythonpath . testmyapp.txt

This will start CherryPy, Selenium’s proxy server and Firefox within which the test case will be run. Easy, elegant and powerful.

Hosting a Django application on a CherryPy server

Recently at work I’ve had the requirement to host a Django application in a CherryPy server. I first looked for various projects I knew were doing just that. Unfortunately, after trying them I was rather disapointed. Their approach is to provide a command similar to the famous Django runserver‘s one but I’ve found it to be more complex than necessary. So I wrote my own module that performs those operations by staying much closer to how CherryPy does work, most specifically by using the process bus coming with CherryPy.

I’m sharing a stripped down version of the module I wrote which shows how one could host a Django application in a CherryPy server. Hopefully this might help some of you.

# Python stdlib imports
import sys
import logging
import os, os.path
 
# Third-party imports
import cherrypy
from cherrypy.process import wspbus, plugins
from cherrypy import _cplogging, _cperror
from django.conf import settings
from django.core.handlers.wsgi import WSGIHandler
from django.http import HttpResponseServerError
 
class Server(object):
    def __init__(self):
        self.base_dir = os.path.join(os.path.abspath(os.getcwd()), "cpdjango")
 
        conf_path = os.path.join(self.base_dir, "..", "server.cfg")
        cherrypy.config.update(conf_path)
 
        # This registers a plugin to handle the Django app
        # with the CherryPy engine, meaning the app will
        # play nicely with the process bus that is the engine.
        DjangoAppPlugin(cherrypy.engine, self.base_dir).subscribe()
 
    def run(self):
        engine = cherrypy.engine
        engine.signal_handler.subscribe()
 
        if hasattr(engine, "console_control_handler"):
            engine.console_control_handler.subscribe()
 
        engine.start()
        engine.block()
 
class DjangoAppPlugin(plugins.SimplePlugin):
    def __init__(self, bus, base_dir):
        """
        CherryPy engine plugin to configure and mount
        the Django application onto the CherryPy server.
        """
        plugins.SimplePlugin.__init__(self, bus)
        self.base_dir = base_dir
 
    def start(self):
        self.bus.log("Configuring the Django application")
 
        # Well this isn't quite as clean as I'd like so
        # feel free to suggest something more appropriate
        from cpdjango.settings import *
        app_settings = locals().copy()
        del app_settings['self']
        settings.configure(**app_settings)
 
        self.bus.log("Mounting the Django application")
        cherrypy.tree.graft(HTTPLogger(WSGIHandler()))
 
        self.bus.log("Setting up the static directory to be served")
        # We server static files through CherryPy directly
        # bypassing entirely Django
        static_handler = cherrypy.tools.staticdir.handler(section="/", dir="static",
                                                          root=self.base_dir)
        cherrypy.tree.mount(static_handler, '/static')
 
class HTTPLogger(_cplogging.LogManager):
    def __init__(self, app):
        _cplogging.LogManager.__init__(self, id(self), cherrypy.log.logger_root)
        self.app = app
 
    def __call__(self, environ, start_response):
        """
        Called as part of the WSGI stack to log the incoming request
        and its response using the common log format. If an error bubbles up
        to this middleware, we log it as such.
        """
        try:
            response = self.app(environ, start_response)
            self.access(environ, response)
            return response
        except:
            self.error(traceback=True)
            return HttpResponseServerError(_cperror.format_exc())
 
    def access(self, environ, response):
        """
        Special method that logs a request following the common
        log format. This is mostly taken from CherryPy and adapted
        to the WSGI's style of passing information.
        """
        atoms = {'h': environ.get('REMOTE_ADDR', ''),
                 'l': '-',
                 'u': "-",
                 't': self.time(),
                 'r': "%s %s %s" % (environ['REQUEST_METHOD'], environ['REQUEST_URI'], environ['SERVER_PROTOCOL']),
                 's': response.status_code,
                 'b': str(len(response.content)),
                 'f': environ.get('HTTP_REFERER', ''),
                 'a': environ.get('HTTP_USER_AGENT', ''),
                 }
        for k, v in atoms.items():
            if isinstance(v, unicode):
                v = v.encode('utf8')
            elif not isinstance(v, str):
                v = str(v)
            # Fortunately, repr(str) escapes unprintable chars, \n, \t, etc
            # and backslash for us. All we have to do is strip the quotes.
            v = repr(v)[1:-1]
            # Escape double-quote.
            atoms[k] = v.replace('"', '\\"')
 
        try:
            self.access_log.log(logging.INFO, self.access_log_format % atoms)
        except:
            self.error(traceback=True)
 
if __name__ == '__main__':
    Server().run()

You can find the code along side a minimal Django application showing how this works here (BSD licence). I used Django 1.3 to generate a default project but the code above works well with older version of Django.

Edit 16/03/2012: Thanks to Damien Tougas, I’ve wrapped up a better recipe for hosting a Django application into a CherryPy application server.