back to index

pypi2nix reborn

Filed under:

A tool that generates nix expressions from python's requirements.txt, buildout.cfg or directly from setup.py. If you are a python developer help me test it.

In the recent years pypi2nix tool went over many iterations. Quite few approaches were tested, and for the last year all pieces finally came together. I can say personally that for the past 6 months I'm a happy pypi2nix user. I finally came around to polish some rough edges and write this blogpost.

Currently I'm looking for python developers in Nix community to give it a try and report bugs, file feature requests or ask questions if something is unclear.

What is pypi2nix?

pypi2nix is tool that tries to generate nix expressions from your projects requirements.txt, buildout.cfg and setup.py.

Python packaging is not the simplest thing you can read all about in one place. It is years of poorly documented work and somehow it all works (well at least most of the time). pypi2nix is never going to be a tool that will work 100% of the time, but in the worse case it will get you pretty close and leave you with only few lines of manual work.

An important thing to keep in mind is that pypi2nix is not a tool that will automate generating pkgs/top-level/python-packages.nix for nixpkgs repository. pypi2nix should be used on project basis (similar to how cabal2nix works). Maybe this will change in the future, but for now this is the current scope of the project.

How do you install it?

pypi2nix was just pushed to nixpkgs master and should take some time until it lands in your channels. You can install it directly from master branch on Github.

% git clone https://github.com/garbas/pypi2nix
% cd pypi2nix
% nix-env -iA build."x86_64-linux" -f release.nix

Once pypi2nix gets built by hydra you can also install it via nix-env command:

% nix-env -iA nixos.pypi2nix

or if you are using Nix on non-NixOS system

% nix-env -iA nixpkgs.pypi2nix

If you want to start contributing to pypi2nix look no further then running nix-shell:

% nix-shell

and you can start hacking on the code.

How do you use it?

It is very common for python project to have requirements.txt file which lists projects dependencies.

For the sake of this blogpost lets create an example requirements.txt

% echo "requests"   >  requirements.txt
% echo "pyramid"    >> requirements.txt
% echo "lxml"       >> requirements.txt

As you can see above I created a requirements.txt file where I specified 3 dependencies. To generate nix expressions we have to run:

% pypi2nix -r requirements.txt -E "libxslt libxml2"
...

Because lxml depends on libxslt and libxml2 we needed to declare them as build input (-E options) in order to install it.

For those new to python packaging world keep in minf that we need to install a package in order to know which are its dependencies. Crazy right!? Well that is how python works.

Above command created 3 files:

  • requirements_generated.nix - A list of generated nix expressions for all packages listed in requirements.txt and all their dependencies. An example of generated expression

    { pkgs, python, commonBuildInputs ? [], commonDoCheck ? false }:
    
    self: {
    
      ...
    
      "Babel" = python.mkDerivation {
        name = "Babel-2.3.4";
        src = pkgs.fetchurl {
          url = "https://files.pythonhosted.org/packages/.../Babel-2.3.4.tar.gz";
          sha256= "...";
        };
        doCheck = commonDoCheck;
        buildInputs = commonBuildInputs;
        propagatedBuildInputs = [
          self."pytz"
        ];
        meta = {
          homepage = "";
          license = lib.bsdOriginal;
          description = "Internationalization utilities";
        };
      };
    
      ...
    
    }
    
  • requirements_override.nix - It is an empty list of overrides that you can write if you are not happy with what pypi2nix generated. This file only gets created if it does not exist yet. More on this later on.

  • requirements.nix - A file that glues together requirements_generated.nix and requirements_override.nix. Also in this file a new python nix functions are implemented. More on this later on.

To build pyramid, lxml and requests do:

% nix-build requirements.nix -A pkgs.pyramid -A pkgs.lxml -A pkgs.requests

Or to build python interpreter with all of the above packages

% nix-build requirements.nix -A interpreter
% ./result/bin/python -c "import pyramid; import lxml; import requests"

Or to enter development environment with all of the above packages

% nix-shell requirements.nix -A interpreter
(nix-shell) % python -c "import pyramid; import lxml; import requests"

By default python 2.7 is selected, but you can choose other python version by specifying -V option

% pypi2nix -r requirements.txt -E "libxslt libxml2" -V "3.5"
...
% nix-shell requirements.nix -A interpreter
(nix-shell) % python3 -c "import pyramid; import lxml; import requests"

All python version in nixpkgs are available in pypi2nix as well

% pypi2nix --help
...
Options:
   ...
   -V, --python-version [2.7|3.5|3.4|3.3|pypy|2.6|3.2]
                                  Provide which python version we build for.
                                  [required]

You can find few more examples in examples folder.

What to do when nix-build fails?

It might (and probably will) happen that some packages will fail to build. There are milion of reasons why some packages will fail to build. It would be foolish trying to solve and accommodate every possible scenario. Instead, when things go south, tools to override generated expressions are provided.

Initially a file with suffix _override.nix is generated with an empty list of overrides.

As an example to show how this overriding works, let us enable tests for lxml library from previous example. A requirements_override.nix file would then look like

{ pkgs, python }:

self: super: {

  "lxml" = python.overrideDerivation super."lxml" (old: {
    doCheck = true;
  });

}

After the change you can continue to build packages as shown above. If you rerun pypi2nix -r requirements.txt you will see that requirements_override.nix does not get overwritten.

Above example gives you all the flexibility to override any existing expression as well as adding new (manually written) expressions.

Why are generated expressions different then the one used in nixpkgs?

If you look at generated nix expressions in requirements_generated.nix and if you packaged python packages with nix in the past you will see that the functions used there are a bit different.

With pypi2nix I also took the freedome to explore how could we have nicer developer experience when working with python and nix. Nothing is set in stone, so I invite you to open an issue if you disagree with the the direction taken.

Current limitations

Current limitations of pypi2nix tool that are going to be fixed in the future are:

  • not possible to specify multiple requirements.txt files (#34)
  • not working with buildout.cfg files (#31)
  • not working with on setup.py files (#32)
  • tests for generated packages are disabled by default for now (#35)
  • requires packages on pypi to have a tarball/zip release. some packages only publish in egg/wheel format (#26)

Let me know if I'm missing some crutial things that I just can not see.

What does the future hold?

pypi2nix is just a first stepping stone on the roadmap to package python packages on PyPI.

End goal for me is that an average python developer (or a consumer of python packages) would be able to depend on any python package without writing a single line of nix code.

Next thing towards that goal I am going to work on is to create a repository of a smaller set automatically generated nix expressions from PyPi. Not sure where the experiment will take me, but I have used similar approach in few previous projects.

I already started a repository called nixpkgs-python where I plan to continue this work, in case you are interested in following/joining this effort.

Convinced to give it a try?

Help me test pypi2nix and provide me with failing requirements.txt. And if it works please give me a shout over twitter. It is nice to know that it is working for somebody.