python package pinning: pip-tools and dephash
discovering pip tools
Last week, I wrote a long blog draft about python package pinning. Then I found this. It's well written, and covers many of the points I wanted to make. The author perfectly summarizes the divide between package development and package deployment to production:
So I dug deeper.
pip-tools #303 perfectly describes the problem I was trying to solve:
Given a minimal dependency list,
- generate a full, expanded dependency list, and
- pin that full dependency list by version and hash.
pip-tools #303 and upstreaming seemed like the ideal solution.
However, pip-tools is broken with pip 8.1.2. The Python Packaging Authority (PyPA) states that pip's public API is the CLI, and
pip-tools could potentially break with every new pip patch release. This is solvable by either using pypa/packaging directly, or switching
pip-tools to use the CLI. That's considerably more work than just integrating hashing capability into
pip-tools. ([EDIT] pip-tools now works with pip 8.1.2, but shoehorning hashes into it is a non-trivial task. I do hope someone tackles it though.)
But I had already whipped up a quick'n'dirty python script that used the pip CLI. (I had assumed that bypassing the internal API was a hack, but evidently this is the supported way of doing things.) So, back to the original blog post, but much shorter:
dephash gen takes a minimal requirements file, and generates an expanded dependency list, pinned by version and hash.
$ cat requirements-dev.txt requests arrow $ dephash gen requirements-dev.txt > requirements-prod.txt $ cat requirements-prod.txt # Generated from dephash.py + hashin.py arrow==0.8.0 \ --hash=sha512:b6c01970d408e1169d042f593859577... python-dateutil==2.5.3 \ --hash=sha512:413b935321f0a65fd8e8ba49990acd5... \ --hash=sha512:d8e28dad57ea85663962f4518faea0e... \ --hash=sha512:107ff2eb6f0715996061262b70844ec... requests==2.10.0 \ --hash=sha512:e5b7d20c4d692b2655c13fa177b8462... \ --hash=sha512:05c6a1a742d31511ca4d3f39534e3e0... six==1.10.0 \ --hash=sha512:a41b40b720c5267e4a47ffb98cdc792... \ --hash=sha512:9a53b7bc8f7e8b358c930eaecf91cc5...
Developers can work against
requirements-dev.txt, with the latest available dependencies. At the same time, production can be pinned against specific package versions+hashes for stability and security.
dephash outdated PATH checks whether
PATH contains outdated packages.
PATH can be a requirements file or virtualenv.
$ cat requirements-outdated.txt six==1.9.0 $ dephash outdated requirements-outdated.txt Found outdated packages in requirements-outdated.txt: six (1.9.0) - Latest: 1.10.0 [wheel]
$ virtualenv -q venv $ venv/bin/pip install -q -r requirements-outdated.txt $ dephash outdated venv Found outdated packages in venv: six (1.9.0) - Latest: 1.10.0 [wheel]
This just uses
pip list --outdated on the backend. I'm tentatively thinking a whitelist of known-outdated dependencies might help here, but I haven't written it yet.
I still think the glorious future involves fixing pip-tools #303 and getting
pip-tools pointed at a supported pypa API. And/or getting
pip-tools upstreamed into
pip. But in the meantime, there's
(I'm leaving my package vendoring musings and python package wishlist for future blogpost(s).)tags: