How to build a multi architecture container using buildx

11 min read

Docker has the ability to handle multi architecture containers: Using the same container image and tag we can deploy it on multiple architectures such as Intel and ARM. Since a docker container is composed of multiple layers it will just use one or another depending on the architecture we are running it. From the user perspective there's no difference on it's usage, but how do we build them?

To be able to create multi architecture containers from a Dockerfile we will have to use docker 19.03+ and enabling the experimental features by editing the /etc/docker/daemon.json as follows:

{
  "experimental": true
}

Then we will create a new builder using docker buildx create specifying the list of plaftforms we want to be able to use:

$ docker buildx create --platform linux/amd64,linux/arm64,linux/arm/v7 --use
cranky_ishizaka

We can check the list of available builders using docker buildx ls:

$ docker buildx ls
NAME/NODE          DRIVER/ENDPOINT             STATUS   PLATFORMS
cranky_ishizaka *  docker-container                     
  cranky_ishizaka0 unix:///var/run/docker.sock inactive linux/amd64*, linux/arm64*, linux/arm/v7*
default            docker                               
  default          default                     running  linux/amd64, linux/386

Then we will have to make sure we have the appropriate emulators correcty installed on the system. The easiest way of doing so is by using tonistiigi/binfmt as follows:

$ docker run --privileged --rm tonistiigi/binfmt --install arm64,arm
Unable to find image 'tonistiigi/binfmt:latest' locally
latest: Pulling from tonistiigi/binfmt
9e0174275344: Pull complete 
f163282b5573: Pull complete 
Digest: sha256:f52cfb6019e8c8d12b13093fd99c2979ab5631c99f7d8b46d10f899d0d56d6ab
Status: Downloaded newer image for tonistiigi/binfmt:latest
2021/06/09 16:31:01 installing: arm64 OK
2021/06/09 16:31:01 installing: arm OK
{
  "supported": [
    "linux/amd64",
    "linux/arm64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "qemu-aarch64",
    "qemu-arm"
  ]
}

We can check whether it is working by running a container using one of the target architectures, for example, for arm64 we can use arm64v8/alpine:

$ docker run --rm arm64v8/alpine uname -a
Unable to find image 'arm64v8/alpine:latest' locally
latest: Pulling from arm64v8/alpine
595b0fe564bb: Pull complete 
Digest: sha256:8f18fae117ec6e5777cc62ba78cbb3be10a8a38639ccfb949521abd95c8301a4
Status: Downloaded newer image for arm64v8/alpine:latest
Linux fa7c06650014 5.8.0-50-generic #56~20.04.1-Ubuntu SMP Mon Apr 12 21:46:35 UTC 2021 aarch64 Linux

Same for armv7:

$ docker run --rm arm32v7/alpine uname -a
Unable to find image 'arm32v7/alpine:latest' locally
latest: Pulling from arm32v7/alpine
e160e00eb35d: Pull complete 
Digest: sha256:9663906b1c3bf891618ebcac857961531357525b25493ef717bca0f86f581ad6
Status: Downloaded newer image for arm32v7/alpine:latest
Linux 2975b7153aa0 5.8.0-50-generic #56~20.04.1-Ubuntu SMP Mon Apr 12 21:46:35 UTC 2021 armv7l Linux

To be able to build a multi architecture container from a Dockerfile the command is not much different than the one we are used to: We just need to use buildx by invoking it using docker buildx build and then set the target platforms we want to use with the --platform option. Oddly enough, the build status appears on the top of the output instead of at the end:

$ docker buildx build --platform linux/arm/v7,linux/arm64,linux/amd64 -t test .
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load 
[+] Building 48.9s (25/25) FINISHED                                                                                                                                                                                
 => [internal] load build definition from Dockerfile                                                                                                                                                          2.8s
 => => transferring dockerfile: 32B                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                             2.9s
 => => transferring context: 2B                                                                                                                                                                               0.0s
 => [linux/arm/v7 internal] load metadata for docker.io/library/python:3.8-alpine                                                                                                                             1.7s
 => [linux/amd64 internal] load metadata for docker.io/library/python:3.8-alpine                                                                                                                              1.7s
 => [linux/arm64 internal] load metadata for docker.io/library/python:3.8-alpine                                                                                                                              1.7s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                                                                                 0.0s
 => [internal] load build context                                                                                                                                                                             0.3s
 => => transferring context: 280B                                                                                                                                                                             0.0s
 => [linux/amd64 1/6] FROM docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                        0.8s
 => => resolve docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                                    0.7s
 => [linux/arm/v7 1/6] FROM docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                       1.3s
 => => resolve docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                                    1.2s
 => [linux/arm64 1/6] FROM docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                        1.7s
 => => resolve docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                                    1.6s
 => CACHED [linux/amd64 2/6] WORKDIR /code                                                                                                                                                                    0.0s
 => CACHED [linux/amd64 3/6] RUN pip install gunicorn                                                                                                                                                         0.0s
 => CACHED [linux/amd64 4/6] COPY requirements.txt .                                                                                                                                                          0.0s
 => CACHED [linux/amd64 5/6] RUN pip install -r requirements.txt                                                                                                                                              0.0s
 => CACHED [linux/amd64 6/6] COPY app /code/app                                                                                                                                                               0.0s
 => CACHED [linux/arm/v7 2/6] WORKDIR /code                                                                                                                                                                   0.0s
 => [linux/arm/v7 3/6] RUN pip install gunicorn                                                                                                                                                              14.5s
 => CACHED [linux/arm64 2/6] WORKDIR /code                                                                                                                                                                    0.0s
 => [linux/arm64 3/6] RUN pip install gunicorn                                                                                                                                                               14.2s
 => [linux/arm/v7 4/6] COPY requirements.txt .                                                                                                                                                                3.6s
 => [linux/arm64 4/6] COPY requirements.txt .                                                                                                                                                                 4.0s 
 => [linux/arm/v7 5/6] RUN pip install -r requirements.txt                                                                                                                                                   20.0s 
 => [linux/arm64 5/6] RUN pip install -r requirements.txt                                                                                                                                                    22.7s 
 => [linux/arm/v7 6/6] COPY app /code/app                                                                                                                                                                     1.7s 
 => [linux/arm64 6/6] COPY app /code/app                                                                                                                                                                      1.2s 

If there's something wrong with the setup we can end up with some nasty results that makes it hard to understand what's going on, for example:

error: failed to solve: rpc error: code = Unknown desc = executor failed running [/dev/.buildkit_qemu_emulator /bin/sh -c pip install gunicorn]: exit code: 1

Doesn't tell much about what's going on, right? For this specific message, what's going on is that the underlying qemu is not setup correctly, so we can fix it using tonistiigi/binfmt. The full output is:

$ docker buildx build --platform linux/arm/v7,linux/arm64,linux/amd64 -t test .
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load 
[+] Building 20.4s (18/24)                                                                                                                                                                                         
 => [internal] load build definition from Dockerfile                                                                                                                                                          2.1s
 => => transferring dockerfile: 32B                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                             2.2s
 => => transferring context: 2B                                                                                                                                                                               0.0s
 => [linux/amd64 internal] load metadata for docker.io/library/python:3.8-alpine                                                                                                                              0.9s
 => [linux/arm/v7 internal] load metadata for docker.io/library/python:3.8-alpine                                                                                                                             3.1s
 => [linux/arm64 internal] load metadata for docker.io/library/python:3.8-alpine                                                                                                                              0.9s
 => [internal] load build context                                                                                                                                                                             1.4s
 => => transferring context: 280B                                                                                                                                                                             0.0s
 => [linux/arm64 1/6] FROM docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                        2.3s
 => => resolve docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                                    1.1s
 => [linux/arm/v7 1/6] FROM docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                       2.3s
 => => resolve docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                                    1.1s
 => [linux/amd64 1/6] FROM docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                        2.4s
 => => resolve docker.io/library/python:3.8-alpine@sha256:9e296fa6f65f947a62398b32b5a54fdfd3f04f3a2c58584172bbe988f794a898                                                                                    1.1s
 => CACHED [linux/arm64 2/6] WORKDIR /code                                                                                                                                                                    0.0s
 => ERROR [linux/arm64 3/6] RUN pip install gunicorn                                                                                                                                                         12.5s
 => CACHED [linux/arm/v7 2/6] WORKDIR /code                                                                                                                                                                   0.0s
 => ERROR [linux/arm/v7 3/6] RUN pip install gunicorn                                                                                                                                                        12.6s
 => CACHED [linux/amd64 2/6] WORKDIR /code                                                                                                                                                                    0.0s
 => CACHED [linux/amd64 3/6] RUN pip install gunicorn                                                                                                                                                         0.0s
 => CACHED [linux/amd64 4/6] COPY requirements.txt .                                                                                                                                                          0.0s
 => CACHED [linux/amd64 5/6] RUN pip install -r requirements.txt                                                                                                                                              0.0s
 => CACHED [linux/amd64 6/6] COPY app /code/app                                                                                                                                                               0.0s
------                                                                                                                                                                                                             
 > [linux/arm64 3/6] RUN pip install gunicorn:                                                                                                                                                                     
#8 7.794 ERROR: Exception:                                                                                                                                                                                         
#8 7.794 Traceback (most recent call last):                                                                                                                                                                        
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in _main                                                                                                     
#8 7.794     status = self.run(options, args)                                                                                                                                                                      
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 204, in wrapper                                                                                                    
#8 7.794     return func(self, options, args)                                                                                                                                                                      
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 269, in run                                                                                                       
#8 7.794     session = self.get_default_session(options)                                                                                                                                                           
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 77, in get_default_session                                                                                         
#8 7.794     self._session = self.enter_context(self._build_session(options))                                                                                                                                      
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 87, in _build_session
#8 7.794     session = PipSession(
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/session.py", line 275, in __init__
#8 7.794     self.headers["User-Agent"] = user_agent()
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/session.py", line 132, in user_agent
#8 7.794     linux_distribution = distro.linux_distribution()  # type: ignore
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 125, in linux_distribution
#8 7.794     return _distro.linux_distribution(full_distribution_name)
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 681, in linux_distribution
#8 7.794     self.version(),
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 741, in version
#8 7.794     self.lsb_release_attr('release'),
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 903, in lsb_release_attr
#8 7.794     return self._lsb_release_info.get(attribute, '')
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 556, in __get__
#8 7.794     ret = obj.__dict__[self._fname] = self._f(obj)
#8 7.794   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 1014, in _lsb_release_info
#8 7.794     stdout = subprocess.check_output(cmd, stderr=devnull)
#8 7.794   File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
#8 7.794     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#8 7.794   File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
#8 7.794     raise CalledProcessError(retcode, process.args,
#8 7.794 subprocess.CalledProcessError: Command '('lsb_release', '-a')' returned non-zero exit status 1.
#8 7.814 Traceback (most recent call last):
#8 7.814   File "/usr/local/bin/pip", line 8, in <module>
#8 7.814     sys.exit(main())
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 71, in main
#8 7.814     return command.main(cmd_args)
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 104, in main
#8 7.814     return self._main(args)
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 221, in _main
#8 7.814     self.handle_pip_version_check(options)
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 143, in handle_pip_version_check
#8 7.814     session = self._build_session(
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 87, in _build_session
#8 7.814     session = PipSession(
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/session.py", line 275, in __init__
#8 7.814     self.headers["User-Agent"] = user_agent()
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/session.py", line 132, in user_agent
#8 7.814     linux_distribution = distro.linux_distribution()  # type: ignore
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 125, in linux_distribution
#8 7.814     return _distro.linux_distribution(full_distribution_name)
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 681, in linux_distribution
#8 7.814     self.version(),
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 741, in version
#8 7.814     self.lsb_release_attr('release'),
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 903, in lsb_release_attr
#8 7.814     return self._lsb_release_info.get(attribute, '')
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 556, in __get__
#8 7.814     ret = obj.__dict__[self._fname] = self._f(obj)
#8 7.814   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 1014, in _lsb_release_info
#8 7.814     stdout = subprocess.check_output(cmd, stderr=devnull)
#8 7.814   File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
#8 7.814     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#8 7.814   File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
#8 7.814     raise CalledProcessError(retcode, process.args,
#8 7.814 subprocess.CalledProcessError: Command '('lsb_release', '-a')' returned non-zero exit status 1.
------
------
 > [linux/arm/v7 3/6] RUN pip install gunicorn:
#21 9.954 ERROR: Exception:
#21 9.954 Traceback (most recent call last):
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
#21 9.954     status = self.run(options, args)
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 204, in wrapper
#21 9.954     return func(self, options, args)
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 269, in run
#21 9.954     session = self.get_default_session(options)
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 77, in get_default_session
#21 9.954     self._session = self.enter_context(self._build_session(options))
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 87, in _build_session
#21 9.954     session = PipSession(
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/session.py", line 275, in __init__
#21 9.954     self.headers["User-Agent"] = user_agent()
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/session.py", line 132, in user_agent
#21 9.954     linux_distribution = distro.linux_distribution()  # type: ignore
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 125, in linux_distribution
#21 9.954     return _distro.linux_distribution(full_distribution_name)
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 681, in linux_distribution
#21 9.954     self.version(),
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 741, in version
#21 9.954     self.lsb_release_attr('release'),
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 903, in lsb_release_attr
#21 9.954     return self._lsb_release_info.get(attribute, '')
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 556, in __get__
#21 9.954     ret = obj.__dict__[self._fname] = self._f(obj)
#21 9.954   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 1014, in _lsb_release_info
#21 9.954     stdout = subprocess.check_output(cmd, stderr=devnull)
#21 9.954   File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
#21 9.954     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#21 9.954   File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
#21 9.954     raise CalledProcessError(retcode, process.args,
#21 9.954 subprocess.CalledProcessError: Command '('lsb_release', '-a')' returned non-zero exit status 1.
#21 9.975 Traceback (most recent call last):
#21 9.975   File "/usr/local/bin/pip", line 8, in <module>
#21 9.975     sys.exit(main())
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 71, in main
#21 9.975     return command.main(cmd_args)
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 104, in main
#21 9.975     return self._main(args)
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 221, in _main
#21 9.975     self.handle_pip_version_check(options)
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 143, in handle_pip_version_check
#21 9.975     session = self._build_session(
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 87, in _build_session
#21 9.975     session = PipSession(
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/session.py", line 275, in __init__
#21 9.975     self.headers["User-Agent"] = user_agent()
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/session.py", line 132, in user_agent
#21 9.975     linux_distribution = distro.linux_distribution()  # type: ignore
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 125, in linux_distribution
#21 9.975     return _distro.linux_distribution(full_distribution_name)
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 681, in linux_distribution
#21 9.975     self.version(),
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 741, in version
#21 9.975     self.lsb_release_attr('release'),
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 903, in lsb_release_attr
#21 9.975     return self._lsb_release_info.get(attribute, '')
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 556, in __get__
#21 9.975     ret = obj.__dict__[self._fname] = self._f(obj)
#21 9.975   File "/usr/local/lib/python3.8/site-packages/pip/_vendor/distro.py", line 1014, in _lsb_release_info
#21 9.975     stdout = subprocess.check_output(cmd, stderr=devnull)
#21 9.975   File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
#21 9.975     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#21 9.975   File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
#21 9.975     raise CalledProcessError(retcode, process.args,
#21 9.975 subprocess.CalledProcessError: Command '('lsb_release', '-a')' returned non-zero exit status 1.
------
Dockerfile:6
--------------------
   4 |     
   5 |     # GUNICORN - not an actual dependency
   6 | >>> RUN pip install gunicorn
   7 |     
   8 |     COPY requirements.txt .
--------------------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/dev/.buildkit_qemu_emulator /bin/sh -c pip install gunicorn]: exit code: 1

Posted on 10/06/2021

Categories