Python packaging is a mess — setuptools, easy_install, distribute, pip? — and that mess makes packaging and distributing a library seem intimidating. It turns out it’s really easy, but I’ve yet to find a concise explanation of how to get started. This post is going to fix that. It will quickly walk through the process of packaging a simple library, getting it onto PyPI, and using C extensions in the package. At the bottom you’ll also find links to the key resources I used to figure this out in case you need more detail.

The goal of this post isn’t to explain all the options. Rather, it describes how to build a package which you should be able to build with pip. I’ll briefly try to define terms or explain things like PyPI, but this really expects that you’re already familiar with using packages and now want to create one.

Creating a Basic Package

First, create your basic package layout, including your module:

| setup.py
+ foo/
   | __init__.py
   | code.py

We’re packaging foo — fill in that package with whatever code you like. The only new thing here should be setup.py, which describes the package. Here’s a template:

from setuptools import find_packages, setup
setup(name="foo",
      version="0.1",
      description="A foo utility",
      author="Ewen Cheslack-Postava",
      author_email='me@ewencp.org',
      platforms=["any"],  # or more specific, e.g. "win32", "cygwin", "osx"
      license="BSD",
      url="http://github.com/ewencp/foo",
      packages=find_packages(),
      )

You can fill in more details, but these are sufficient. Notice that the only setting that doesn’t have an obvious value is packages, and it turns out that the packaging system can almost always figure out the right value for us. That’s all there is to it.

To test, use development mode. This links the current directory into your site-packages, where regular packages are installed via pip (e.g. in /usr/local/lib/python2.7/site-packages/ or in a virtualenv). It then works as if it were installed normally, but can be easily disconnected later. Run:

python setup.py develop

and you should now be able to run something like this without an error:

import foo
foo.fn()

This mode is very nice since you can run it once, make modifications, and test without having to re-install the package. When you’re ready to stop developing (e.g. later when you want to actually install the package), run

python setup.py develop --uninstall

Once you’re convinced your package is ready, generate the source tar (“source distribution”):

python setup.py sdist

You’ll find the tar file under the dist/ folder. I highly recommend looking at it to make sure it contains everything you expect to be there is there.

Publishing to PyPI

PyPI is a centralized location that stores information about available packages so you can look them up and easily install them by name. When you run pip install foo, it’s looking up the instructions for installing the package on PyPI. If you register and upload your package to PyPI, others developers can install your package just as easily.

With the package ready, we just need to upload it to PyPI. You need to register a user account on PyPI if you don’t have one.

Next, you need to register your package with PyPI. Again, a setup.py command does just that:

python setup.py register

All the relevant information is pulled from your setup.py. You just need to enter your username and password. This step only reserves the name of the package; no files have been uploaded yet. You could check on PyPI that the package is now there without any files, but let’s just upload the source:

python setup.py sdist upload

That’s it, now running

pip install foo

should install your package. The upload command will upload whatever the current version is, so you can just bump the version number, build the tar, and issue the upload command again to release a new version.

Adding Dependencies

What if you require some other packages? Just specify them as dependencies:

setup(...,
      install_requires=["docutils>=0.3", "setuptools==0.5"],
      )

This is a nice trick if you already have a requirements.txt that specifies the full list of packages for pip to install (optionally with specific versions to use):

setup(...,
      install_requires=[i.strip() for i in open("requirements.txt").readlines()],
      )

Including a C Extension

The package I was building was just a wrapper around some C code. In my case, there wasn’t even any Python code. To include a C extension like this, first create a directory named for the module and put the C source code in it. For example, pyhashxx looks like this:

| setup.py
+ pyhashxx/
   | xxhash.h
   | xxhash.c
   | pycompat.h
   | pyhashxx.c

(Note that often you’ll have the C code in _pyhashxx with a Python wrapper module called pyhashxx that imports and wraps _pyhashxx. I wanted to expose the C module directly without any Python overhead, so I didn’t use this naming scheme.)

Next update your setup.py to add the extension (showing mostly changes here):

from setuptools import find_packages, setup, Extension

pyhashxx = Extension('pyhashxx',
                     sources=['pyhashxx/xxhash.c',
                              'pyhashxx/pyhashxx.c'],
                     depends=['pyhashxx/xxhash.h',
                              'pyhashxx/pycompat.h'])

setup(..., ext_modules=[pyhashxx],)

That should be it – the compilation steps should be taken care of as long as a compiler can be found. If you’re in development mode, you’ll need to run

python setup.py build

to build the extension. If you modify any C files, make sure you re-run this so you’re running the updated code.

Note that some documentation doesn’t cover the very important depends option: if you miss it, these files will not be included in the package. However, it’s easy to miss because the development mode discussed above works in your working directory rather than making a full copy. This is one of the reasons I suggested checking the contents of the tar file before submitting it to PyPI — I screwed up my first version and had to bump the version number because my first version was broken.

Extra Notes

Tests are, you know, good. You can specify the test suite in your setup.py:

setup(...,
      test_suite="tests"
      )

which lets you test easily using

python setup.py test

There are also a lot of tweaks and convenient features you can use when building extensions that aren’t mentioned here. This just gets you up and running. If you’re serious about developing extensions, you should invest some time reading the resources below.

Resources

• Building and Distributing Packages with Distribute – This is the key page. It covers most of what’s included here, but in a lot more detail. In being comprehensive, it’s not very practical for helping get you started with packaging.

• How To Package Your Python Code – Tutorial-like description of packaging, like this page, but in a lot more detail.

• Python Packaging: Hate, hate, hate everywhere – want to know how we got into the current packaging mess?

• Current State of Packaging – Graphical version of the previous, showing how all the tools relate to each other.

Thanks

Thanks to Jeff Terrace for reviewing this post.