Pulling images from ECR on Kubernetes

Kubernetes ECR HTTP 403 Forbidden no basic auth credentials

2 min read | by Jordi Prats

If you are getting the HTTP 403 (Forbidden) error or the error message no basic auth credentials when trying to pull an image from ECR you are most likely doing so without logging into it first.

This is going to happen if we are running a Kubernetes cluster on EC2 instances instead of using EKS. To be able to authenticate before pulling images on Kubernetes we need to use the imagePullSecrets attribute that's going to reference the secret containing the credentials.

To get the ECR credentials (assuming our instance profile allow us to do it) we can use the following AWS CLI command:

aws ecr get-login-password

We can use the AmazonEC2ContainerRegistryReadOnly managed policy to generically allow pull access to ECR but we can also narrow it down to a specific image using a custom policy.

Next, to create a secret with these credentials to pull from the account 123456789123 on the region eu-west-1 we can create it as follows:

kubectl create secret docker-registry ecr \
  --docker-server 123456789123.dkr.ecr.eu-west-1.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password) -n kube-system

On the deployment side we'll have to reference this secret using imagePullSecrets as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-ecr
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: demo-ecr
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: demo-ecr
    spec:
      containers:
      - image: 123456789123.dkr.ecr.eu-west-1.amazonaws.com/pet2cattle/demo:v2.4.1
        name: demo-ecr
      imagePullSecrets:
      - name: ecr

These credentials are going to expire after 12 hours, so we'll need to find a way of updating them, for example a cronjob on one of the nodes or even a Kubernetes cronjob.

On the other hand, if we are using EKS we won't need to worry about these details since the cluster will take care of getting the credentials for us without needing to explicitly set imagePullSecrets

Posted on 18/05/2022

Categories