Running rootless containers with nerdctl

nerdctl rootless containers docker

6 min read | by Jordi Prats

It's no secret that docker comes stuffed with many options that many of us don't need. This is why on servers we can find containerd instead of a fully featured docker. Despite that, the real deal breaker is that whatever we are running, we are going to do it with root privileges. We can check this by running the following container:

$ docker run -v /etc:/itc -it --rm alpine sleep 24

And then looking for the process on the host

$ ps auxf

root        1307  0.0  0.1 2084820 46676 ?       Ssl  11:36   0:04 /usr/bin/containerd
root       66978  0.0  0.0 709860  6120 ?        Sl   05:12   0:00  \_ containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/50cf9789d0e68949d1cf79462956bde98b46a4616e8
b81977d8c89d2af9c34e7 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root       66996  2.0  0.0   1588     4 pts/0    Ss+  05:12   0:00      \_ sleep 24

Is it possible to run rootless containers? Is there an alternative to docker?

We could check the docker documentation for a rootless setup but, instead of properly setting up a rootless docker, we can switch to nerdctl. And that's not just because it has a catchy name, nerdctl is a Docker-compatible CLI for containerd. As a matter of fact, it's name comes from it: containerd

contai NERDCTL

With nerdctl we are going to be able to seamlessly run rootless containers, although to be able to do so we'll need to install a few things a root

First, we will have to install the rootlesskit, on it's website we will find all the details but in a nutshell it's going to be:

Set the following sysctl settings as /etc/sysctl.d/99-rootless.conf:

kernel.unprivileged_userns_clone=1

And then load them using:

$ sudo sysctl --system

We'll also need uidmap and slirp4netns, so on a Debian-based distribution we can install it as follows:

sudo apt-get install -y uidmap
sudo apt-get install slirp4netns

Then we can install the rootlesskit downloading it's latest release from github and installing it into the local bin directory of whatever user we want to use:

$ tar xzf rootlesskit-x86_64.tar.gz -C $HOME/.local/bin

Once this bit is installed we can proceed installing nerdctl from it's github releases page as well:

$ tar xzvf nerdctl-0.16.1-linux-amd64.tar.gz  -C $HOME/.local/bin

The last thing we will need to install are the CNI plugins into /opt/cni/bin. We can also download them from it's github releases page:

sudo mkdir /opt/cni/bin -p
$ sudo tar xzf cni-plugins-linux-amd64-v1.0.1.tgz -C /opt/cni/bin

Finally we can install the rootless containerd running the following:

$ containerd-rootless-setuptool.sh install
[INFO] Checking RootlessKit functionality
[INFO] Checking cgroup v2
[WARNING] Enabling cgroup v2 is highly recommended, see https://rootlesscontaine.rs/getting-started/common/cgroup2/ 
[INFO] Checking overlayfs
[INFO] Requirements are satisfied
[INFO] Creating "/home/pet2cattle/.config/systemd/user/containerd.service"
[INFO] Starting systemd unit "containerd.service"
+ systemctl --user start containerd.service
+ sleep 3
+ systemctl --user --no-pager --full status containerd.service
 containerd.service - containerd (Rootless)
     Loaded: loaded (/home/pet2cattle/.config/systemd/user/containerd.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-02-07 06:04:57 CET; 3s ago
   Main PID: 29624 (rootlesskit)
     CGroup: /user.slice/user-1000.slice/user@1000.service/containerd.service
             ├─29624 rootlesskit --state-dir=/run/user/1000/containerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --copy-up=/var/lib --propagation=rslave /home/pet2cattle/.local/bin/containerd-rootless.sh
             ├─29634 /proc/self/exe --state-dir=/run/user/1000/containerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --copy-up=/var/lib --propagation=rslave /home/pet2cattle/.local/bin/containerd-rootless.sh
             ├─29651 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 29634 tap0
             └─29659 containerd

feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.154737145+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.154821946+01:00" level=info msg="Start subscribing containerd event"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.154897593+01:00" level=info msg="Start recovering state"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.154979404+01:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155003418+01:00" level=info msg="Start event monitor"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155017947+01:00" level=info msg=serving... address=/run/containerd/containerd.sock
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155022518+01:00" level=info msg="Start snapshots syncer"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155037164+01:00" level=info msg="Start cni network conf syncer"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155041900+01:00" level=info msg="containerd successfully booted in 0.609421s"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155043447+01:00" level=info msg="Start streaming server"
+ systemctl --user enable containerd.service
Created symlink /home/pet2cattle/.config/systemd/user/default.target.wants/containerd.service  /home/pet2cattle/.config/systemd/user/containerd.service.
[INFO] Installed "containerd.service" successfully.
[INFO] To control "containerd.service", run: `systemctl --user (start|stop|restart) containerd.service`
[INFO] To run "containerd.service" on system startup automatically, run: `sudo loginctl enable-linger pet2cattle`
[INFO] ------------------------------------------------------------------------------------------
[INFO] Use `nerdctl` to connect to the rootless containerd.
[INFO] You do NOT need to specify $CONTAINERD_ADDRESS explicitly.

Once it's ready we will be able to run container using nerdctl:

$ nerdctl run -it --rm alpine
/ # cat /etc/alpine-release 
3.15.0
/ # 

If we run a command (sleep 24h) like we did with docker and look for the actual process on the host server, we will be able to see that it's no longer running as root:

$ ps auxf
(...)
pet2cattle      2314  0.0  0.0  22352 11768 ?        Ss   11:37   0:00 /lib/systemd/systemd --user
pet2cattle      2315  0.0  0.0 169536  3900 ?        S    11:37   0:00  \_ (sd-pam)
pet2cattle      2320  7.8  0.0 2420528 22836 ?       S<sl 11:37  19:40  \_ /usr/bin/pulseaudio --daemonize=no --log-target=journal
pet2cattle      2322  0.0  0.0 728088 28708 ?        SNsl 11:37   0:00  \_ /usr/libexec/tracker-miner-fs
pet2cattle      2326  0.0  0.0   8104  5256 ?        Ss   11:37   0:10  \_ /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
(...)
pet2cattle     29624  0.0  0.0 709708  7900 ?        Ssl  06:04   0:00  \_ rootlesskit --state-dir=/run/user/1000/containerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --copy-up=/var/lib --propagation=rslave /home/pet2cattle/.local/bin/containerd-rootless.sh
pet2cattle     29634  0.0  0.0 709708  7076 ?        Sl   06:04   0:00  |   \_ /proc/self/exe --state-dir=/run/user/1000/containerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --copy-up=/var/lib --propagation=rslave /home/pet2cattle/.local/bin/containerd-rootless.sh
pet2cattle     29659  0.0  0.1 2158616 58340 ?       Sl   06:04   0:06  |   |   \_ containerd
pet2cattle     29651  0.0  0.0   5540  3216 ?        S    06:04   0:00  |   \_ slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 29634 tap0
pet2cattle     61387  0.0  0.0 1217648 19260 ?       SNl  07:44   0:00  \_ /usr/lib/x86_64-linux-gnu/tumbler-1/tumblerd
pet2cattle     61615  0.2  0.0 711664  8720 ?        Sl   07:45   0:00  \_ /usr/bin/containerd-shim-runc-v2 -namespace default -id 6f5b878ff06aed5ec4486ac6b5d4673bd22c20646dd7553cb2c6d9d68f1f2e61 -address /run/containerd/containerd.sock
pet2cattle     61635  0.0  0.0   1664  1108 pts/0    Ss   07:45   0:00      \_ /bin/sh
pet2cattle     61949  0.0  0.0   1584     4 pts/0    S+   07:46   0:00          \_ sleep 24h

As a consequence, there are going to be commands that we won't be able to run:

/ # id
uid=0(root) gid=0(root) groups=1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
/ # mknod null c 1 3
mknod: null: Operation not permitted

On the bright side, containers won't be entitled to do whatever they want with the system. For example, when using docker there's nothing preventing you from mounting the host's /etc directory into a container. Once you have it mounted, you are accessing this file using docker's permissions (aka root):

$ docker run -v /etc:/itc -it --rm alpine
/ # ls /itc/shadow
/itc/shadow
/ # head -n1 /itc/shadow
root:!:18569:0:99999:7:::
/ # 

By using rootless containers with nerdctl we won't be able to read files on the host's file system that our user doesn't already have access to:

$ nerdctl run -v /etc:/itc -it --rm alpine
/ # ls /itc/shadow
/itc/shadow
/ # head -n1 /itc/shadow
head: /itc/shadow: Permission denied
/ # 

Posted on 08/02/2022

Categories