I realized I have never really sat down and learned systemd. I was mostly exposed to it via NixOS when writing NixOS modules and time came to dig a bit deeper. Systemd man pages are invaluable source of information. Initially they might be a bit overwhelming, but soon you start to appreciate the long explanation.
journalctl became one of the tools I can not really work without. To be worry free when it comes to logging and to not be a awk/grep wizard when it comes to accessing logs is something everybody new to systemd will appreciate.
Some (hopefully) useful commands I wrote down during Lennart's journald presentation:
- journalctl -r - show logs in reverse order
- journalctl -b - show logs since last boot
- journalctl -k - show kernel logs
- journalctl -p warning - show logs with warning priority
- journalctl -p error - show logs with error priority
- journalctl --since=2016-08-01 - show logs since
- journalctl --until=2016-08-03 - show logs until
- journalctl --until=today - show logs until midnight today
- journalctl --since=yesterday - show logs since yesterday midnight
- journalctl --since=-2week - show logs for last 2 weeks
- journalctl -u <unit-name> - show logs of certain unit
- journalctl /dev/sda - show kernel message of device
- journalctl -o json - show logs in json format
And you can mix more or less or of the above options.
To serve journald events over the networks there is systemd-journal-gatewayd service which is disabled by default.
Every log message can now provide MESSAGE_ID (generated via journalctl --new-id128) to be able to use journal's catalog capabilities. If this gains traction, it could really help to have better documentation with every error that shows up in journal.
Currently we can limit globally how much space logs will take. Option to have this per service would be very nice and might happen in future.
Containers were the hot topic at systemd conference. Not about their current usage, but more or less looking into the future and how containers might look.
- Vincent Batts: What’s next for containers?
- Alban Crequy, Luca BRUNO: Securing services and containers: rkt meets systemd
- Lennart Poettering: State of the Union/Portable Services
- Djalal Harouni: Apps sandboxing in systemd
Apart from great talks I was not aware that there are commands to pull / import / export containers from command line
% machinectl pull-tar <url> % machinectl pull-raw <url> % machinectl import-tar <name> <file> % machinectl import-raw <name> <file> % machinectl export-tar <name> <file> % machinectl export-raw <name> <file>
Not something revolutionary, but this commands can be useful to quickly share images with coworkers.
Lennart also presented another tool called mkosi which is aiming to build legacy free OS images. You can consider it as a replacement for deboostrap and dnf. I wonder if this is a place were Nix could be used, since we already have a way of creating images of non NixOS distribution.
All the negativity that some people have towards systemd, got me convinced that systemd is one big monolith and that it can not be stripped down to the bare minimum. Last and this year show us that systemd on embedded devices is growing and while there still are problems those problems are possible to solve.
- Gustavo Sverzut Barbieri: Demystifying systemd for embedded systems
- Jens Georg: Automotive startup and device management
- Michael: Reconciling Systemd and Customer Requirements
Listening to all the talks about embedded systems got me thinking what kind of build systems do they use and could Nix be a possible solution. Knowing the strong points of Nix (the build tool), reproducibility, composability, etc... must be features that would get some developers in embedded world interested. But to unlock this space for Nix, a basic support for ARM should be there, maybe just in a form of binary channel for a small subset of packages.
One of pain points of my current setup (no desktop manager like kde/gnome/xfce, only i3 window manager) is that I have to manually mount every USB stick. I was not aware that systemd could also handle this.
/dev/sdb1 /mnt/usb auto noauto,x-systemd.automount 0 2
Looks like systemd community is really tackling the hard problems. While there is a networkd which manages networking (there was also a talk about it: Lennart Poettering, Tom Gundersen: What you didn't know about networkd) there is also work in progress to have have a utility to manage wireless networks, Marcel Holtmann: New Wireless Daemon for Linux.
I think the last area where Linux feels painful would be Printing. systemd-printing anybody? :)
Systemd unit files allow us to specify services in a declarative way. With systemd growing in scope we are more and more seeing a need for a better way of managing services. We have seen to talk about configuration management.
We have seen an very nice presentation of a even more awesome tool called Mgmt (James (purpleidea) Shubin: Next Generation Config Mgmt). While still a bit unpolished you can easily see being it superior to many existing configuration tools out there.
Completely different approach to Configuration management took NixOS. And I had an honor to speak about it (NixOS - Reproducible Linux Distribution built around systemd). I tried my best to explain the core principles of how Nix works, but sadly I was short in time to show all the demos. Still I have learned few things when presenting Nix/NixOS:
- Don't mention functional, ignore it
- There is no Nix Expression Language, but a JSON-like syntax
- Not everybody sees the benefits of reproducibility in first 5 seconds, show examples
- Nix is not for everybody and for every use case