Running rootless containers with nerdctl
6 min read | by Jordi Prats
It's no secret that docker comes stuffed with many options that many of us don't need. This is why on servers we can find containerd instead of a fully featured docker. Despite that, the real deal breaker is that whatever we are running, we are going to do it with root privileges. We can check this by running the following container:
$ docker run -v /etc:/itc -it --rm alpine sleep 24
And then looking for the process on the host
$ ps auxf
root 1307 0.0 0.1 2084820 46676 ? Ssl 11:36 0:04 /usr/bin/containerd
root 66978 0.0 0.0 709860 6120 ? Sl 05:12 0:00 \_ containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/50cf9789d0e68949d1cf79462956bde98b46a4616e8
b81977d8c89d2af9c34e7 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root 66996 2.0 0.0 1588 4 pts/0 Ss+ 05:12 0:00 \_ sleep 24
Is it possible to run rootless containers? Is there an alternative to docker?
We could check the docker documentation for a rootless setup but, instead of properly setting up a rootless docker, we can switch to nerdctl. And that's not just because it has a catchy name, nerdctl is a Docker-compatible CLI for containerd. As a matter of fact, it's name comes from it: containerd
With nerdctl we are going to be able to seamlessly run rootless containers, although to be able to do so we'll need to install a few things a root
First, we will have to install the rootlesskit, on it's website we will find all the details but in a nutshell it's going to be:
Set the following sysctl settings as /etc/sysctl.d/99-rootless.conf:
kernel.unprivileged_userns_clone=1
And then load them using:
$ sudo sysctl --system
We'll also need uidmap and slirp4netns, so on a Debian-based distribution we can install it as follows:
sudo apt-get install -y uidmap
sudo apt-get install slirp4netns
Then we can install the rootlesskit downloading it's latest release from github and installing it into the local bin directory of whatever user we want to use:
$ tar xzf rootlesskit-x86_64.tar.gz -C $HOME/.local/bin
Once this bit is installed we can proceed installing nerdctl from it's github releases page as well:
$ tar xzvf nerdctl-0.16.1-linux-amd64.tar.gz -C $HOME/.local/bin
The last thing we will need to install are the CNI plugins into /opt/cni/bin. We can also download them from it's github releases page:
sudo mkdir /opt/cni/bin -p
$ sudo tar xzf cni-plugins-linux-amd64-v1.0.1.tgz -C /opt/cni/bin
Finally we can install the rootless containerd running the following:
$ containerd-rootless-setuptool.sh install
[INFO] Checking RootlessKit functionality
[INFO] Checking cgroup v2
[WARNING] Enabling cgroup v2 is highly recommended, see https://rootlesscontaine.rs/getting-started/common/cgroup2/
[INFO] Checking overlayfs
[INFO] Requirements are satisfied
[INFO] Creating "/home/pet2cattle/.config/systemd/user/containerd.service"
[INFO] Starting systemd unit "containerd.service"
+ systemctl --user start containerd.service
+ sleep 3
+ systemctl --user --no-pager --full status containerd.service
● containerd.service - containerd (Rootless)
Loaded: loaded (/home/pet2cattle/.config/systemd/user/containerd.service; disabled; vendor preset: enabled)
Active: active (running) since Mon 2022-02-07 06:04:57 CET; 3s ago
Main PID: 29624 (rootlesskit)
CGroup: /user.slice/user-1000.slice/user@1000.service/containerd.service
├─29624 rootlesskit --state-dir=/run/user/1000/containerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --copy-up=/var/lib --propagation=rslave /home/pet2cattle/.local/bin/containerd-rootless.sh
├─29634 /proc/self/exe --state-dir=/run/user/1000/containerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --copy-up=/var/lib --propagation=rslave /home/pet2cattle/.local/bin/containerd-rootless.sh
├─29651 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 29634 tap0
└─29659 containerd
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.154737145+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.154821946+01:00" level=info msg="Start subscribing containerd event"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.154897593+01:00" level=info msg="Start recovering state"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.154979404+01:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155003418+01:00" level=info msg="Start event monitor"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155017947+01:00" level=info msg=serving... address=/run/containerd/containerd.sock
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155022518+01:00" level=info msg="Start snapshots syncer"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155037164+01:00" level=info msg="Start cni network conf syncer"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155041900+01:00" level=info msg="containerd successfully booted in 0.609421s"
feb 07 06:04:58 croscat.pet2cattle.com containerd-rootless.sh[29659]: time="2022-02-07T06:04:58.155043447+01:00" level=info msg="Start streaming server"
+ systemctl --user enable containerd.service
Created symlink /home/pet2cattle/.config/systemd/user/default.target.wants/containerd.service → /home/pet2cattle/.config/systemd/user/containerd.service.
[INFO] Installed "containerd.service" successfully.
[INFO] To control "containerd.service", run: `systemctl --user (start|stop|restart) containerd.service`
[INFO] To run "containerd.service" on system startup automatically, run: `sudo loginctl enable-linger pet2cattle`
[INFO] ------------------------------------------------------------------------------------------
[INFO] Use `nerdctl` to connect to the rootless containerd.
[INFO] You do NOT need to specify $CONTAINERD_ADDRESS explicitly.
Once it's ready we will be able to run container using nerdctl:
$ nerdctl run -it --rm alpine
/ # cat /etc/alpine-release
3.15.0
/ #
If we run a command (sleep 24h) like we did with docker and look for the actual process on the host server, we will be able to see that it's no longer running as root:
$ ps auxf
(...)
pet2cattle 2314 0.0 0.0 22352 11768 ? Ss 11:37 0:00 /lib/systemd/systemd --user
pet2cattle 2315 0.0 0.0 169536 3900 ? S 11:37 0:00 \_ (sd-pam)
pet2cattle 2320 7.8 0.0 2420528 22836 ? S<sl 11:37 19:40 \_ /usr/bin/pulseaudio --daemonize=no --log-target=journal
pet2cattle 2322 0.0 0.0 728088 28708 ? SNsl 11:37 0:00 \_ /usr/libexec/tracker-miner-fs
pet2cattle 2326 0.0 0.0 8104 5256 ? Ss 11:37 0:10 \_ /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
(...)
pet2cattle 29624 0.0 0.0 709708 7900 ? Ssl 06:04 0:00 \_ rootlesskit --state-dir=/run/user/1000/containerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --copy-up=/var/lib --propagation=rslave /home/pet2cattle/.local/bin/containerd-rootless.sh
pet2cattle 29634 0.0 0.0 709708 7076 ? Sl 06:04 0:00 | \_ /proc/self/exe --state-dir=/run/user/1000/containerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --copy-up=/var/lib --propagation=rslave /home/pet2cattle/.local/bin/containerd-rootless.sh
pet2cattle 29659 0.0 0.1 2158616 58340 ? Sl 06:04 0:06 | | \_ containerd
pet2cattle 29651 0.0 0.0 5540 3216 ? S 06:04 0:00 | \_ slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 29634 tap0
pet2cattle 61387 0.0 0.0 1217648 19260 ? SNl 07:44 0:00 \_ /usr/lib/x86_64-linux-gnu/tumbler-1/tumblerd
pet2cattle 61615 0.2 0.0 711664 8720 ? Sl 07:45 0:00 \_ /usr/bin/containerd-shim-runc-v2 -namespace default -id 6f5b878ff06aed5ec4486ac6b5d4673bd22c20646dd7553cb2c6d9d68f1f2e61 -address /run/containerd/containerd.sock
pet2cattle 61635 0.0 0.0 1664 1108 pts/0 Ss 07:45 0:00 \_ /bin/sh
pet2cattle 61949 0.0 0.0 1584 4 pts/0 S+ 07:46 0:00 \_ sleep 24h
As a consequence, there are going to be commands that we won't be able to run:
/ # id
uid=0(root) gid=0(root) groups=1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
/ # mknod null c 1 3
mknod: null: Operation not permitted
On the bright side, containers won't be entitled to do whatever they want with the system. For example, when using docker there's nothing preventing you from mounting the host's /etc directory into a container. Once you have it mounted, you are accessing this file using docker's permissions (aka root):
$ docker run -v /etc:/itc -it --rm alpine
/ # ls /itc/shadow
/itc/shadow
/ # head -n1 /itc/shadow
root:!:18569:0:99999:7:::
/ #
By using rootless containers with nerdctl we won't be able to read files on the host's file system that our user doesn't already have access to:
$ nerdctl run -v /etc:/itc -it --rm alpine
/ # ls /itc/shadow
/itc/shadow
/ # head -n1 /itc/shadow
head: /itc/shadow: Permission denied
/ #
Posted on 08/02/2022