back to index

Updating your Nix sources

Filed under:

Updating sources (versions, revisions, tags) can be a tedius manual work as a package maintainer in nixpkgs. Just looking at current nixpkgs repository we can see that majority of pull requests are about updating sources of packages. How can this be improved?

It feels a bit tedious that we (package maintainers of nixpkgs) are still updating version of packages manually. Especially in such a small community, we should be really careful where we use our resources.

In this blog post I will take you on a tour of things I learned in a project I am working on with Release Engineering team at Mozilla, mozilla-releng/services, and how we are continuously updating our nix sources.

Extending nixpkgs

Many times I was asked questions how to work efficiently with upstream nixpkgs and how to work with private nix package set.

For each project I usually create a private set of nix expressions. This could look like:

{ pkgs ? import <nixpkgs> {}
}:

let
  custom_pkgs = {
    inherit pkgs;
    packageA = ...;
    packageB = ...;
  };
in custom_pkgs

Our nix expression set is called custom_pkgs and it is a function which takes nixpkgs as an argument.

Pinning nixpkgs

In the past I already wrote about why pinning nixpkgs matters. To recap: We always want to have a stable development environment. Importing <nixpkgs> would depend on system's nixpkgs which is different from machine to machine. To pin down nixpkgs let us change previous example:

let
  _pkgs = import <nixpkgs> {};
in
{ pkgs ? import (_pkgs.fetchFromGitHub { owner = "NixOS";
                                         repo = "nixpkgs-channels";
                                         rev = "...";
                                         sha256 = "...";
                                       }) {}
}:

let
  custom_pkgs = {
    inherit pkgs;
    packageA = ...;
    packageB = ...;
  };
in custom_pkgs

We still depend on system's <nixpkgs>, but only to provide us with _pkgs.fetchFromGitHub function. We then pin NixOS/nixpkgs-channels repository to specific revision. I chose nixpkgs-channels repository, since that means I will also get binaries and wont have to recompile too often.

Update runner script

mozilla-releng/services's update runner script can be found in nix/update.nix. What this script does is, it checks which package has update attribute and then loops over and executes every update script. A minimal example would look like:

let
  _pkgs = import <nixpkgs> {};
in
{ pkgs ? import (_pkgs.fetchFromGitHub { owner = "NixOS";
                                         repo = "nixpkgs-channels";
                                         rev = "...";
                                         sha256 = "...";
                                       }) {}
}:

let
  packagesWith = name: attrs: ...; # function which searches attrs values
                                   # and checks for name attribute.
  custom_pkgs = import ./default.nix { inherit pkgs; };
  packages = packagesWith "update" attrs;
in pkgs.stdenv.mkDerivation {
  name = "update-releng-services";
  buildCommand = ''
    echo "+--------------------------------------------------------+"
    echo "| Not possible to update repositories using \`nix-build\`. \|"
    echo "|         Please run \`nix-shell update.nix\`.             \|"
    echo "+--------------------------------------------------------+"
    exit 1
  '';
  shellHook = ''
    export HOME=$PWD
    echo "Updating packages ..."
    ${builtins.concatStringsSep "\n\n" (
        map (pkg: ''echo ' - ${(builtins.parseDrvName pkg.name).name}';
                    ${pkg.update};
                  '') packages
    )}
    echo ""
    echo "Packages updated!"
    exit
  '';
}

If above script is ran with nix-build it will raise an error, saying this script can only be ran with nix-shell. Update scripts will need access to internet, and this is the reason we must run it with nix-shell.

When we run nix-shell update.nix we can see that packagesWith function currently does not find any package with update attribute, since we did not define any.

Update script

Lets first create an update script for tracking nixos-unstable branch for NixOS/nixpks-channels repository on github. Implementation of such script can be found here, updateFromGitHub.

Now lets use updateFromGitHub function and add update attribute to nixpkgs attribute in our custom_pkgs.

pkgs = pkgs // {
   update = releng_pkgs.lib.updateFromGitHub {
     owner = "garbas";
     repo = "nixpkgs";
     branch = "python-srcs";
     path = "nix/nixpkgs.json";
   };
 };
 let
   _pkgs = import <nixpkgs> {};
 in
 { pkgs ? import (_pkgs.fetchFromGitHub
                     (_pkgs.lib.importJSON ./nixpkgs.json)) {}
 }:

 let
   updateFromGitHub = ...;
   custom_pkgs = {
     pkgs = pkgs // {
       update = updateFromGitHub {
         owner = "NixOS";
         repo = "nixpkgs-channels";
         branch = "nixos-unstable";
         path = "nixpkgs.json";
       };
     };
     packageA = ...;
     packageB = ...;
   };
 in custom_pkgs

As you can see above our nixpkgs update script stores owner, repo, rev and sha256 to a nixpkgs.json file in a format which we also read and pass to fetchFromGitHub to be used to initialize custom_pkgs.

An example of update script for a python project, which uses pypi2nix, would look like this:

packageA = pkgs.stdenv.mkDerivation {
  ...
  passthru.update = writeScript "update-${name}" ''
    pushd src/packageA
    ${pkgs.pypi2nix}/bin/pypi2nix -v \
     -V 3.5 \
     -E "postgresql libffi openssl" \
     -r requirements.txt \
     -r requirements-dev.txt
    popd
  '';
  ...
}

Automation with Travis CI

Now that we can manually run update script we need to run it on a daily basis. Luckily for us Travis supports this.

  1. Enable Travis for your project.
  2. Ask Travis for cron support.
  3. Create .travis.yml such as:
language: nix
script:
- if [ "$TRAVIS_EVENT_TYPE" == "cron" ]; then
    nix-shell update.nix --pure;
  fi
- nix-build default.nix
before_deploy:
- eval "$(ssh-agent -s)"
- chmod 600 $TRAVIS_BUILD_DIR/deploy_rsa
- ssh-add $TRAVIS_BUILD_DIR/deploy_rsa
after_success:
- if [[ -n `git diff --exit-code` ]]; then
    git config user.name 'travis';
    git config user.email 'you@email';
    git stash
    git checkout -b result-$TRAVIS_BRANCH origin/$TRAVIS_BRANCH
    git pull
    git stash apply
    git add .
    git commit -m "Travis update [skip ci]";
    git push git@github.com:<owner>/<repo>.git HEAD:$TRAVIS_BRANCH;
  fi
  1. Create deploy_rsa using following commands:
% ssh-keygen -t rsa -b 4096 -C 'travis/<owner>/<repo>' -f ./deploy_rsa
% travis encrypt-file deploy_rsa --add
  1. Allow deploy_rsa.pub to push to your repository.

Add content of deploy_rsa.pub to Authorized keys for your github repository and make sure it has Push permission.

  1. Commit everything and watch updates coming in.
% git add deploy_rsa.enc deploy_rsa.pub .travis
% git commit -c "Travis will make us all lazy."
% git push

Conclusion

It does take quite some time to setup all of this, but benefits can be seen within a week once first (auto-)commits are coming in.

It is important to look at your project as a living system where latest version should be automatically picked, something we are used to from traditional packages managers. With Nix it is not possible to depend on future releases.

With above setup we can get best of both worlds. Despite constantly updating to latest versions, we don't update versions if build (including tests) is not passing. This way we have flexibility of traditional package managers and robustness of Nix.

Best of both worlds.

With large number of packages or projects a setup like this will save us a lot of time. Now imagine how much time we could save if we have similar setup for nixpkgs.

Let me know what you think. Nix out.