back to index

Reproducible development environments

Filed under:

Look at your past experience and give a good estimate how much time did you spend fixing bugs? Which part of fixing bugs was the hardest?

First thing before fixing a bug is being able to reproduce it. Usually once you are able to reproduce the bug solution comes easy. Probably this does not come as a surprise and we all experienced this. But actually reproducing a bug very often requires ninja skills or just a lot of time that you could spend playing with your dog or cat or goldfish.

Creating a reproducible builds with Nix is what Nix is all about. But it gets a bit tricy to create a trully reproducible build.

What many of us - Nix users - forget is that Nix is first a build tool which also happens to be a package manager. What we forget is that we can leverage all of the features that come with Nix (the packages manager), eg. atomicity, rollbacks,... when setting up out a development environment.

Below lines require a bit more knowledge of Nix (the language). If you have not learn it, please take a look at the official Nix manual and a very nice introduction to Nix by James Fisher.

Example project

For the sake of writing this blog post lets create an example project and call it ProjectA.

% mkdir -p ProjectA/bin
% cd ProjectA
% echo "#\!/bin/sh" > bin/serve
% echo "python -m SimpleHTTPServer 8000" >> bin/serve
% chmod +x bin/serve
% git init
% git add bin/serve
% git commit -m "Initial commit."

We just created a simple bash script which start a http server on port 8000, Then we initialized a git repository and committed our bin/serve script. Nothing special, just something for out example.

Everything starts simple

To nixify the project we usually create Nix expression at the root of the project repository and call it default.nix. For our project our default.nix would look like:

{ }:

let
  pkgs = import <nixpkgs> { };
in
  pkgs.stdenv.mkDerivation {
    name = "ProjectA-1.0.0";
    src = ./.;
    buildInputs = [ pkgs.python ];
    installPhase= ''
      sed -i -e "s|python|`which python`" bin/serve
      mkdir -p $out/bin
      cp bin/* $out/bin
    '';
  }

For those familiar with Nix, above expression shouldn't be anything hard to understand what it does.

To enter development environment you would simple have to type nix-shell and voila, all the tools needed to develope ProjectA are there. This is python in out example.

Troubles in paradise

What I really liked about above instructions is that all you have to remember is to run nix-shell. You even don't have to understand Nix. You can simple type nix-shell and you will be ready to develop.

Now above code has one drawback. Once somebody clones your repository and runs nix-shell they might end up with different environment. This is because they might have a different version of nixpkgs on their system.

One way to solve this would be to use -I nixpkgs=path/to/nixpkgs or NIX_PATH variable and make developer responsible of making sure his development environment is being created against correct version of nixpkgs.

But that is just calling for troubles. What I would like is developers to just type nix-shell and not have to mess with anything else. Nix should be able to do this, right?

Making it truly reproducible

While working at RhodeCode, I have come up with a solution which made it possible to just type nix-shell and it will build against identical nixpkgs as other developers in the team.

Trick is pinning nixpkgs inside default.nix itself.

I am sure I am not the first one using this feature of Nix, but what I have not seen it being mentioned in this context.

Lets take a look at adjusted default.nix.

let
  _nixpkgs = import <nixpkgs> { };
in

{ nixpkgs ? (import _nixpkgs.fetchFromGitHub { owner = "NixOS"; repo = "nixpkgs"; rev = ...; sha256 = ...; })
}:

let
  pkgs = if nixpkgs == null then _nixpkgs else nixpkgs;
in
  pkgs.stdenv.mkDerivation {
    name = "ProjectA-1.0.0";
    src = ./.;
    buildInputs = [ pkgs.python ];
    installPhase= ''
      sed -i -e "s|python|`which python`" bin/serve
      mkdir -p $out/bin
      cp bin/* $out/bin
    '';
  }

We did quite a few things:

  • We added an argument to a default.nix called nixpkgs.
  • Default value of nixpkgs argument is a know-to-work version of nixpkgs.
  • Instead of cloning and keeping nixpkgs version manually, we use system nixpkgs (_nixpkgs in our script) to clone it for us.
  • In case nixpkgs argument is null we fallback to what defined by system (via -I or NIX_PATH). In case of hydra we would set nixpkgs argument to null and let hydra manage it.

Conclusion

Pinning to specific version of nixpkgs made it possible for developers not familiar with Nix to simple know one command nix-shell to bootstrap their environment.

It made possible to always have a working environment identical to that revision regardless of the system version of nixpkgs. This means can travel back in time of project history in your VCS and reproduce exact environment for each revision.

We did not sacrifice flexibility over simplicity. Running against custom version of nixpkgs would be as simple as running nix-shell --arg nixpkgs /absolute/path/to/nixpkgs.

To let hydra build against different version of nixpkgs we only need to set nixpkgs argument to null.

Hope this helps to make it easier to share your projects.

For comments or corrections please send me a tweet or better, write a blog post in response and then send me a tweet :)