Tag Archives: acceptance testing

Robot Framework and Sphinx: A suitable toolset for your specification by example

At work, we have been using Robot Framework for all kinds of tests for a few years now and it’s proven to be the good choice. Robot Framework’s simple syntax and grammar does not scare testers away (usually). At the same time, its design makes it easy to support complex use cases as well as simple ones through the power of the Python programming language.

One blind spot however, in my opinion anyway, is the way Robot Framework let you document your tests. It provides a section for this, with basic HTML support but it has always felt limited and not really friendly.

Luckily, in the recent releases, the Robot Framework developers have provided a built-in support for reStructuredText. Not that the documentation section supports this syntax, but instead, you can embed Robot Framework tests into a reStructuredText document, and therefore into Sphinx as well.

The gain isn’t so much visible in the Robot Framework reports since the reStructuredText sections won’t appear in those, but it means you can generate HTML documents which embed executable tests. Fans of doctests will be in known territory.

I think this is a powerful combination as it bridges the tests with the specifications and ensure they are both kept locally at the same place, imrpoving their chance to stay synchronised.  In my mind, it provides a great framework to follow the Specification by Example that Gojko Adzic described so eloquently.

Here is a simple:

Finally, a related powerful extension provides a simple mechanism to include Robot Framework tests into Sphinx documentation. We use it extensively at work as we wanted to keep our tests outside in distinct files without losing the ability to see them embedded into the generated HTML documentation.

“Robot Framework Test Automation” book review

From time to time PacktPub will request a book review of one of their Python-related titles. This time around it was regarding their “Robot Framework Test Automation” book they recently released. Since I’ve been using this awesome acceptance testing tool at work for more than two years, I was happy to comply.

In a nutshell, Robot Framework provides a great interface that acts as the middle-man between variour stakeholders. Indeed, tests are written in plain text (though other formats are supported, I never use them) with a rather minimal set of rules making it (almost) straightforward to read even by non-technical persons. The dirty technical details being hidden away and implemented in Python and executable in one of the various Python VM (CPython, Jython, IronPython are supported out of the box).

Most of the time, the basics of the Robot Framework data model and workflow can be taught in a couple of hours. However, being efficient with it will take a little more time. Still, people don’t have to learn a complete programming language (Python) itself and that’s a relief meaning they are happy to work with Robot Framework sometimes cumbersome syntax.

In spite of having a rather extensive documentation available online, the project did lack a good, straight to the point summary that takes you by the hand. Moreover, the documentation’s style of the project is fairly dry and Unix-style making it tedious to browse sometimes. Still, the content is there and it rarely failed me. With that said, having a friendly book on the subject is a great thing. Kudos to PacktPub. Now about the book…

The good

The book provides an introduction to the tool, its most common usages and even tries to guide you getting more from it. It’s a short book, 83 pages, that will not bore you with complex details. In other words, it’s a good companion of the online documentation if you start with Robot Framework.

Sumit Bisht, the author, does a good job keeping a neutral point of view in regards to how you should use Robot Framework. Indeed, depending on your software under test, you might want to have a more data-oriented approach (ala fitness), a behavior-driven testing approach or even a more assert-oriented style. Not many software can deal with all of them equally and it depends also on how testing is perceived in your organisation. Robot Framework can cope with all of them.

The bad

Though I could understand it’s only an introduction, it feels like some concepts are not properly explored. The idea behind keywords, the internal data model, dynamic libraries, etc. In other words, you will not really understand the underlying blocks and axioms that are the pedestal of the whole tool, you’ll rather learn the basics of using it. In fact, the only section where the book goes into more technical details (with a good example on using sikuli) will probably confuse you since it failed to properly introduce the principles behind them.

The ugly

There isn’t anything particulary that bad with this book, again it should be considered as a friendly introduction. I do not agree with a few minor points Sumit makes but they hardly matter and aren’t wrong anyway, just a matter of opinion. Note also that the book lacks examples a couple of times where it would have mattered but I don’t believe this makes the book any less useful.

The only thing that annoys me really is that PacktPub book’s layout still looks so unprofesionnal. They should really make an effort as the code is, most of the time, too hard to read (actually on this one item, it wasn’t that bad).

Final note

I think this book is ideal if you are about to start with Robot Framework as it will speed up the basics. If you’re already used to the tool, I am not sure it will help very much.

 

 

 

Acceptance testing a CherryPy application with Robot Framework

I recently received the Python Testing Cookbook authored by Greg L. Turnquist and was happy to read about recipes on acceptance testing using Robot Framework. We’ve been using this tool at work for a few weeks now with great results. Greg shows how to test a web application using the Selenium Library extension for Robot Framework and I thought it’d be fun to demonstrate how to test a CherryPy application following his recipe. So here we go.

First some requirements:

$ mkvirtualenv --distribute --no-site-packages --unzip-setuptools acceptance
(acceptance)$ pip install cherrypy
(acceptance)$ pip install robotframework
(acceptance)$ pip install robotframework-seleniumlibrary

Let’s define a simple CherryPy application, which displays a input text where to type a message. When the submit button is pressed, the message is sent to the server and returned as-is. Well it’s an echo message really.

import cherrypy
 
__all__ = ['Echo']
 
class Echo(object):
    @cherrypy.expose
    def index(self):
        return """<html>
<head><title>Robot Framework Test for CherryPy</title></head>
<body>
<form method="post" action="/echo">
<input type="text" name="message" />
<input type="submit" />
</form>
</body>
</html>"""
 
    @cherrypy.expose
    def echo(self, message):
        return message
 
if __name__ == '__main__':
    cherrypy.quickstart(Echo())

Save the code above in a module named myapp.py

Next, we create an extension to Robot Framework that will manage CherryPy. Save the following in a module CherryPyLib.py. It’s important to respect that name since Robot Framework expects the module and its class to match in names.

import imp
import os, os.path
 
import cherrypy
 
class CherryPyLib(object):
    def setup_cherrypy(self, conf_file=None):
        """
        Configures the CherryPy engine and server using
        the built-in 'embedded' environment mode.
 
        If provided, `conf_file` is a path to a CherryPy
        configuration file used in addition.
        """
        cherrypy.config.update({"environment": "embedded"})
        if conf_file:
            cherrypy.config.update(conf_file)            
 
    def start_cherrypy(self):
        """
        Starts a CherryPy engine.
        """
        cherrypy.engine.start()
 
    def exit_cherrypy(self):
        """
        Terminates a CherryPy engine.
        """
        cherrypy.engine.exit()
 
    def mount_application(self, appmod, appcls, directory=None):
        """
        Mounts an application to be tested. `appmod` is the name
        of a Python module containing `appcls`. The module is
        looked for in the given directory. If not provided, we use
        the current one instead.
        """
        directory = directory or os.getcwd()
        file, filename, description = imp.find_module(appmod, [directory])
        mod = imp.load_module(appmod, file, filename, description)
        if hasattr(mod, appcls):
            cls = getattr(mod, appcls)
            app = cls()
            cherrypy.tree.mount(app)
        else:
            raise ImportError, "cannot import name %s from %s" % (appcls, appmod)

Note that we start and stop the CherryPy server during the test itself, meaning you don’t need to start it separately. Pure awesomeness.

Finally let’s write a straightforward acceptance test to validate the overall workflow of echoing a message using our little application.

***Settings***
Library	SeleniumLibrary
Library	CherryPyLib
Suite Setup	Start Dependencies
Suite Teardown	Shutdown Dependencies
Test Setup	Mount Application	myapp	Echo

***Variables***
${MSG}	Hello World
${HOST}	http://localhost:8080/

***Test Cases***
Echo ${MSG}
     Open Browser	${HOST}
     Input text		message		${MSG}
     Submit form
     Page Should Contain		${MSG}
     Close All Browsers

***Keywords***
Start Dependencies
    Setup Cherrypy
    Start CherryPy
    Start Selenium Server
    Sleep 	3s

Shutdown Dependencies
    Stop Selenium Server
    Exit CherryPy

Save the test above into a file named testmyapp.txt. You can finally run the test as follow:

(acceptance)$ pybot --pythonpath . testmyapp.txt

This will start CherryPy, Selenium’s proxy server and Firefox within which the test case will be run. Easy, elegant and powerful.