Python packaging is a mess —
pip? — and that mess makes packaging and
distributing a library seem intimidating. It turns out it’s really
easy, but I’ve yet to find a concise explanation of how to get
started. This post is going to fix that. It will quickly walk
through the process of packaging a simple library, getting it onto
PyPI, and using C extensions in the package. At the bottom you’ll also
find links to the key resources I used to figure this out in case you
need more detail.
The goal of this post isn’t to explain all the options. Rather,
it describes how to build a package which you should be able to build
pip. I’ll briefly try to define terms or explain things like
PyPI, but this really expects that you’re already familiar with using
packages and now want to create one.
Creating a Basic Package
First, create your basic package layout, including your module:
| setup.py + foo/ | __init__.py | code.py
foo — fill in that package with whatever code you
like. The only new thing here should be
setup.py, which describes
the package. Here’s a template:
from setuptools import find_packages, setup setup(name="foo", version="0.1", description="A foo utility", author="Ewen Cheslack-Postava", firstname.lastname@example.org', platforms=["any"], # or more specific, e.g. "win32", "cygwin", "osx" license="BSD", url="http://github.com/ewencp/foo", packages=find_packages(), )
You can fill in more details, but these are sufficient. Notice that
the only setting that doesn’t have an obvious value is
it turns out that the packaging system can almost always figure out
the right value for us. That’s all there is to it.
To test, use development mode. This links the current directory into
site-packages, where regular packages are installed via
(e.g. in /usr/local/lib/python2.7/site-packages/ or in a
virtualenv). It then works as if it were installed normally, but can
be easily disconnected later. Run:
python setup.py develop
and you should now be able to run something like this without an error:
import foo foo.fn()
This mode is very nice since you can run it once, make modifications, and test without having to re-install the package. When you’re ready to stop developing (e.g. later when you want to actually install the package), run
python setup.py develop --uninstall
Once you’re convinced your package is ready, generate the source tar (“source distribution”):
python setup.py sdist
You’ll find the tar file under the
dist/ folder. I highly recommend
looking at it to make sure it contains everything you expect to be
there is there.
Publishing to PyPI
PyPI is a centralized location that stores information about available
packages so you can look them up and easily install them by name. When
pip install foo, it’s looking up the instructions for
installing the package on PyPI. If you register and upload your
package to PyPI, others developers can install your package just as
With the package ready, we just need to upload it to PyPI. You need to register a user account on PyPI if you don’t have one.
Next, you need to register your package with PyPI. Again, a
command does just that:
python setup.py register
All the relevant information is pulled from your
setup.py. You just
need to enter your username and password. This step only reserves the
name of the package; no files have been uploaded yet. You could check
on PyPI that the package is now there without any files, but let’s
just upload the source:
python setup.py sdist upload
That’s it, now running
pip install foo
should install your package. The upload command will upload whatever the current version is, so you can just bump the version number, build the tar, and issue the upload command again to release a new version.
What if you require some other packages? Just specify them as dependencies:
setup(..., install_requires=["docutils>=0.3", "setuptools==0.5"], )
This is a nice trick if you
already have a
requirements.txt that specifies the full list of
pip to install (optionally with specific versions to
setup(..., install_requires=[i.strip() for i in open("requirements.txt").readlines()], )
Including a C Extension
The package I was building was just a wrapper around some C code. In
my case, there wasn’t even any Python code. To include a C extension
like this, first create a directory named for the module and put the C
source code in it. For example,
pyhashxx looks like this:
| setup.py + pyhashxx/ | xxhash.h | xxhash.c | pycompat.h | pyhashxx.c
(Note that often you’ll have the C code in
_pyhashxx with a Python
wrapper module called
pyhashxx that imports and wraps
wanted to expose the C module directly without any Python overhead, so
I didn’t use this naming scheme.)
Next update your
setup.py to add the extension (showing mostly
from setuptools import find_packages, setup, Extension pyhashxx = Extension('pyhashxx', sources=['pyhashxx/xxhash.c', 'pyhashxx/pyhashxx.c'], depends=['pyhashxx/xxhash.h', 'pyhashxx/pycompat.h']) setup(..., ext_modules=[pyhashxx],)
That should be it – the compilation steps should be taken care of as long as a compiler can be found. If you’re in development mode, you’ll need to run
python setup.py build
to build the extension. If you modify any C files, make sure you re-run this so you’re running the updated code.
Note that some documentation doesn’t cover the very important
depends option: if you miss it, these files will not be included
in the package. However, it’s easy to miss because the development
mode discussed above works in your working directory rather than
making a full copy. This is one of the reasons I suggested checking
the contents of the tar file before submitting it to PyPI — I
screwed up my first version and had to bump the version number because
my first version was broken.
Tests are, you know, good. You can specify the test suite in your
setup(..., test_suite="tests" )
which lets you test easily using
python setup.py test
There are also a lot of tweaks and convenient features you can use when building extensions that aren’t mentioned here. This just gets you up and running. If you’re serious about developing extensions, you should invest some time reading the resources below.
Building and Distributing Packages with Distribute – This is the key page. It covers most of what’s included here, but in a lot more detail. In being comprehensive, it’s not very practical for helping get you started with packaging.
How To Package Your Python Code – Tutorial-like description of packaging, like this page, but in a lot more detail.
Python Packaging: Hate, hate, hate everywhere – want to know how we got into the current packaging mess?
Current State of Packaging – Graphical version of the previous, showing how all the tools relate to each other.
Thanks to Jeff Terrace for reviewing this post.