The joy of distributing Python packages for Python 2 and 3

I released ws4py 0.3.4 this weekend and although I had integrated support for Python 2 and 3 for a long time now, I ran into a challenge I had quite missed. Indeed, until now my Python 3 support had mainly been concerned about string handling and various compatibility modules. This has proven to work very well and avoided having to rely on external packages such as six.

However, in the past few weeks and I added asyncio support to ws4py and therefore introduced the newly yield from statement. Of course, this isn’t tolerated by Python 2 which complains with a well deserved SyntaxError.

The issues however is that I wished to distribute the same source code with a single source distribution archive. Initially, I had written a function that was preventing modules using that statement to actually be packaged. However, this was rather daft since that, if it weren’t packaged, it wouldn’t be distributed either. Next, I decided to some of setuptools magic. Well, it didn’t help since that’s not what it’s there for anyhow. At this stage, I should say: Don’t simply copy/paste. It will do no good.

Finally, I opted for a fairly simple solution. I knew that when a package is installed from a distribution packages, it is obviously built first. I therefore had to act after Python modules had been gathered but before they would be built. After briefly browsing through distutils source code, I found where I would perform surgery: the find_package_modules of the distutils.command.build_py.build_py class. The nice aspect of this solution is that the source distribution contains indeed all the modules, whether they aim Python 2 or 3 but it’s only when installed that the appropriate modules will be selected and built.

Am I doing it wrong? Is there a cleaner, nicer more pythonic way? If so, please let me know. If not, I hope this may help others that want a simple solution to handle their Python 2 and 3 modules in a single baseline.

3 thoughts on “The joy of distributing Python packages for Python 2 and 3”

  1. I went the other route [1], [2] and just use py3 code if possible. I had a problem with the test runner trying to import everything so I have this special py.test hook [3].

    I don’t trust the tooling to always do what I expect. For example, your builder won’t be called if you do a “develop” install or use “pip install -e”. Generally I try to avoid putting any serious code in – remember when people were forcing a distribute upgrade from their script ? Horrible times, horrible ideas …

    But you might have a problem I don’t have – users importing the py3 modules directly – getting a SyntaxError is bad, non-obvious behavior. I think if you make proxy modules that the users import and raise ImportError(“Not available on Python 2”) would be good enough.


    1. Hi,

      I’m not sure I follow you with the “develop” or “pip install -e” since you are not building the guilty modules, they are never approached (unless you have broad imports that tend to load all your modules from your top-level __init__, which I usually don’t do).

      With that said, I agree with you that it’s not fun to hack around distutils. It will eventually break something.

  2. I’ve been poking around ws4py, and thinking that it should be trivial to use it as a starting point for CherryPy + Server Sent Events, if only I knew what I was doing. 🙂 It seems like this bit is the critical part:

    “Basically, when the WebSocket handshake is complete, we take over the socket and let CherryPy take back the thread that was associated with the upgrade request.”

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.