back to index

Building ACI images with Nix

Filed under:

There was never really the right time to look at what CoreOS is working on. First systemd conference, that happened last weekend, was a perfect time and place to see how CoreOS and their infrastructure could be mixed with Nix.

First a big thank you to all the organisers of systemd conference. You did an amazing job and I can not wait for 2016 edition. I do wish there would be some place in the program for lightning/ignate talks that are organized ad-hoc, but now I am just being picky. See you in 2016!

rkt containers

rkt containers are build with help of systemd inside and take different approach how to specify containers, then for example Docker. More details you can find in the video from Alban and Jon.

I am holding my judgement of how I feel about whole rkt approach, but I can definetly agree with Alban and Jon that it goes a way to reinvent package management. And in this context I do not mean anything bad with reinventing. On the contrary, it shows that CoreOS developers have the same feeling about the current situation and state of packaging as me.

package managers are broken!

Maybe they did not want to upset anybody and they did not say it out laud but traditional packaging tools are broken. And as traditional I mean packaging tools that are used in more popular linux distributions (apt-get/deb, yum/rpm, brew, emerge, pacman, ...).

It would make this blog post too long to explain everything properly and argue what I actually mean when calling tools broken. But I promise to come back in a 2 week with a blog post only about this. I promise!

Nix and containers

What you can read from Nix's web page is that Nix is a package manager. In NixOS (linux distribution build on top of Nix package manager) we are using systemd and also its containers (systemd-nspawn).

With help of many CoreOS developers at systemd conference hackaton, I have got to a point where I could be dangerous with rkt and acbuild. Thank you all for your patience and to show me how all this tools work.

Building first ACI image with Nix

The following part of the blog post assumes that you already have Nix installed on your system. If not get it from

0. Install rkt and acbuild

You can skip this step if you already have rkt and acbuild installed on your system otherwise run:

% nix-shell -p rkt goPackages.acbuild

(nix-shell) % which rkt

(nix-shell) % which acbuild

Think of nix-shell command as ad-hoc creation of custom shell environment. Like virtualenv is used in Python, but nix-shell can be used for any language on any unix-like system. Cool right?

In our case we want rkt and acbuild tools to be present this ah-hoc shell environment.

1. Creating ah-hoc Nix profile

Profiles and user environments are Nix's mechanism for implementing the ability to allow different users to have different configurations, and to do atomic upgrades and rollbacks. More on Nix profiles.

To install bash and ipython into Nix profile you have to run:

(nix-shell) % nix-env -i bash ipython -p out
(nix-shell) % ls out/bin
bash     ipcluster     ipengine  iptest   ipython-qtconsole  pycolor
bashbug  ipcontroller  iplogger  ipython  irunner            sh

Above we can see that at the current directory, a symlink, out was created, linking to /nix/store/... and all binaries are located inside out/bin folder.

2. Bulding ACI from Nix profile

Below code creates ACI image with every runtime dependency of bash and ipython.

(nix-shell) % acbuild begin
(nix-shell) % acbuild label add arch amd64
(nix-shell) % acbuild label add os linux
(nix-shell) % for i in $(nix-store -qR out/); do sudo acbuild copy $i $i; done
(nix-shell) % acbuild set-name bash-ipython
(nix-shell) % acbuild set-exec `realpath out/bin/bash`
(nix-shell) % acbuild environment add PATH `realpath out`/bin
(nix-shell) % acbuild write image.aci
(nix-shell) % acbuild end

I guess above acbuild commands are more or less self explanatory, except for the nix-store command.

Nix has tools (like nix-store in this case) which can figure out what are runtime dependencies of certain Nix profile. nix-store -qR <derivation> prints out all runtime dependencies of <derivation>.

Exactly the functionality that is needed to create ACI image with everything inside it. Lets look:

(nix-shell) % nix-store -qR out/

3. Run it!

(nix-shell) % sudo rkt run --interactive --insecure-skip-verify ./image.aci
rkt: using image from local store for image name
rkt: using image from file /home/rok/tmp/tmp/image.aci
run: group "rkt" not found, will use default gid when rendering images
bash-4.2# ipython
Python 2.7.2 (default, Mar  2 2012, 13:25:19)
Type "copyright", "credits" or "license" for more information.

IPython 0.11 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: print "IT WORKS!"
Do you really want to exit ([y]/n)? y
bash-4.2# exit

I guess my bash output can speak for itself. rkt brings us to bash from where we can invoke ipython. Lovely!

Or as CoreOS folks would say: SHIP IT!!! :)

Running non secure applications in rkt containers

One more idea of usage that came to my mind is running non-secure applications inside rkt containers. Example of such application would be Skype. It is not that Skype developers are evil, but just knowing that one application is sending data in closed binary protocol out from your laptop might be considered non-secure.

0. Installing Skype

First we will install Skype using Nix, so that it gets stored inside /nix/store/... folder.

% nix-env -i skype
% which skype
% realpath `which skype`

With above command we installed Skype into default user profile (/home/rok/.nix-profile) which is symlinked into /nix/store/....

1. Building new ACI image

This time we will build a more general ACI image. Make sure that you are still in the nix-shell environment that we created before.

(nix-shell) % acbuild begin
(nix-shell) % acbuild label add arch amd64
(nix-shell) % acbuild label add os linux
(nix-shell) % acbuild set-name nixos
(nix-shell) % acbuild environment add DISPLAY unix$DISPLAY
(nix-shell) % acbuild set-exec `realpath out/bin/bash`
(nix-shell) % acbuild mount add nix-store /nix/store
(nix-shell) % acbuild mount add x11socket /tmp/.X11-unix
(nix-shell) % acbuild mount add dbus /etc/machine-id
(nix-shell) % acbuild write nixos.aci
warning: exec command was never set.
(nix-shell) % acbuild end

We prepared mounts for volumes to be mounted. /nix/store for all the nix install applications (you can look at this as /usr). /tmp/.X11-unix socket to have the ability to run the applications that require X11. And last /etc/machine-id so that applications that need dbus can get to it.

2. Running application

Now all that is left is running the application.

(nix-shell) % sudo rkt run \
    --insecure-skip-verify \
    --volume nix-store,kind=host,source=/nix/store \
    --volume x11socket,kind=host,source=/tmp/.X11-unix \
    --volume dbus,kind=host,source=/etc/machine-id \
        ./nixos.aci \
            --exec="`realpath \`which skype\``"
Skype running in rkt container

Final thoughts

There are many things that one could benefit from building ACI images with Nix:

  • smaller size of images because of sharing /nix/store across host and other images
  • reproducible way of building images
  • whole building infrastructure that comes with Nix (Hydra)

Nix (and its cloud deployment tool nixops) on the other side could gain another target environment to deploy to.

Probabaly there is more that these two tools could benefit from each other, by only time will tell.

For comments or corrections please send me a tweet or better write a blog post in response and then send me a tweet :)