Argo Rollouts: Canary deployments
7 min read | by Jordi Prats
A canary deployment is a technique to reduce the risk of introducing a new version of a software application in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure. If any issue is detected on the "canary", the deployment can be stopped, and the rest of the users won't be affected. With Argo Rollouts, we can easily implement this strategy.
Installing Argo Rollouts
First we'll have to make sure we have Argo Rollouts and it's CLI installed:
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
brew install argoproj/tap/kubectl-argo-rollouts
To start using the canary deployment strategy with Argo Rollouts, we need to update the Deployment manifest to use the Rollout resource and set the stategy to canary:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: canary-rollout
spec:
replicas: 10
selector:
matchLabels:
app: canary-rollout
template:
metadata:
labels:
app: canary-rollout
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
strategy:
canary:
maxSurge: '25%'
maxUnavailable: 0
In the previous example we are using the canary strategy but without any additional configuration. This will make it behave like a regular rolling update.
Simple canary deployment
To make it a canary deployment, we'll need to design the steps we want to follow. In a simple canary deployment we can use setWeight and pause to control how we are going to do it:
- setWeight: The percentage of the new version to be deployed.
- pause: The time to wait between steps. We can set a specific duration or wait for a manual resume.
During the rollout, the controller by default will keep the previous version running at it's maximum replicas, and the new version will be scaled up to the desired replicas. If the rollout is successful, the previous version will be scaled down to zero. This is to make sure we can switch back to the previous version in case of any issue without having to wait for the replicas to be scaled up again. If we don't want to use this approach, we can set dynamicStableScale to true so that it will automatically scale down the previous version as it is scaling up the new one.
Let's see an example:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: canary-rollout
spec:
replicas: 10
selector:
matchLabels:
app: canary-rollout
template:
metadata:
labels:
app: canary-rollout
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
env:
- name: APP_VERSION
value: "v2"
strategy:
canary:
steps:
- setWeight: 20
- pause: {}
- setWeight: 50
- pause:
duration: 10m
- setWeight: 90
- pause: {}
In this example we are:
- First, we are going to scale up to 20% of the total replicas, and wait for a manual resume.
- Then we are going to scale up to 50% of the total replicas, and wait for 10 minutes.
- Finally we are going to scale up to 90% of the total replicas and wait for a manual resume again.
We can use the kubectl argo rollouts get rollout command to check the status of the rollout:
$ kubectl argo rollouts get rollout canary-rollout
Name: canary-rollout
Namespace: demo-rollout
Status: ॥ Paused
Message: CanaryPauseStep
Strategy: Canary
Step: 1/6
SetWeight: 20
ActualWeight: 20
Images: nginx:latest (canary, stable)
Replicas:
Desired: 10
Current: 10
Updated: 2
Ready: 10
Available: 10
NAME KIND STATUS AGE INFO
⟳ canary-rollout Rollout ॥ Paused 10m
├──# revision:2
│ └──⧉ canary-rollout-7f9c9956ff ReplicaSet ✔ Healthy 20s canary
│ ├──□ canary-rollout-7f9c9956ff-6bvt9 Pod ✔ Running 10s ready:1/1
│ └──□ canary-rollout-7f9c9956ff-dsg2l Pod ✔ Running 10s ready:1/1
└──# revision:1
└──⧉ canary-rollout-8fc79696d ReplicaSet ✔ Healthy 10m stable
├──□ canary-rollout-8fc79696d-57gz5 Pod ✔ Running 10m ready:1/1
├──□ canary-rollout-8fc79696d-6ghm9 Pod ✔ Running 10m ready:1/1
├──□ canary-rollout-8fc79696d-8fmjd Pod ✔ Running 10m ready:1/1
├──□ canary-rollout-8fc79696d-9dq6r Pod ✔ Running 10m ready:1/1
├──□ canary-rollout-8fc79696d-jqzsp Pod ✔ Running 10m ready:1/1
├──□ canary-rollout-8fc79696d-pz46t Pod ✔ Running 10m ready:1/1
├──□ canary-rollout-8fc79696d-t8k7j Pod ✔ Running 10m ready:1/1
└──□ canary-rollout-8fc79696d-vsmmx Pod ✔ Running 10m ready:1/1
Since we don't have any specific duration for this step, we'll need to resume the rollout manually usign the promote command:
$ kubectl argo rollouts promote canary-rollout
rollout 'canary-rollout' promoted
$ kubectl argo rollouts get rollout canary-rollout
Name: canary-rollout
Namespace: demo-rollout
Status: ॥ Paused
Message: CanaryPauseStep
Strategy: Canary
Step: 3/6
SetWeight: 50
ActualWeight: 50
Images: nginx:latest (canary, stable)
Replicas:
Desired: 10
Current: 10
Updated: 5
Ready: 10
Available: 10
NAME KIND STATUS AGE INFO
⟳ canary-rollout Rollout ॥ Paused 24m
├──# revision:2
│ └──⧉ canary-rollout-7f9c9956ff ReplicaSet ✔ Healthy 14m canary
│ ├──□ canary-rollout-7f9c9956ff-6bvt9 Pod ✔ Running 13m ready:1/1
│ ├──□ canary-rollout-7f9c9956ff-dsg2l Pod ✔ Running 13m ready:1/1
│ ├──□ canary-rollout-7f9c9956ff-4jqrf Pod ✔ Running 4s ready:1/1
│ ├──□ canary-rollout-7f9c9956ff-6d5m4 Pod ✔ Running 4s ready:1/1
│ └──□ canary-rollout-7f9c9956ff-hjrnj Pod ✔ Running 4s ready:1/1
└──# revision:1
└──⧉ canary-rollout-8fc79696d ReplicaSet ✔ Healthy 24m stable
├──□ canary-rollout-8fc79696d-57gz5 Pod ✔ Running 24m ready:1/1
├──□ canary-rollout-8fc79696d-6ghm9 Pod ✔ Running 24m ready:1/1
├──□ canary-rollout-8fc79696d-8fmjd Pod ✔ Running 24m ready:1/1
├──□ canary-rollout-8fc79696d-t8k7j Pod ✔ Running 24m ready:1/1
└──□ canary-rollout-8fc79696d-vsmmx Pod ✔ Running 24m ready:1/1
If we have a duration set, the rollout will automatically resume after the time has passed or we can also make it to continue by using the promote command.
Canary and stable services
If we define the canaryService and stableService, the controller will update the services to select the right set of Pods.
First we'll need to create these services with a generic selector for the Rollout to use:
apiVersion: v1
kind: Service
metadata:
name: rollout-canary
spec:
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: canary-rollout
---
apiVersion: v1
kind: Service
metadata:
name: rollout-stable
spec:
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: canary-rollout
Having the services created, we can how create a new Rollout using these services:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: canary-rollout
spec:
replicas: 10
selector:
matchLabels:
app: canary-rollout
template:
metadata:
labels:
app: canary-rollout
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
env:
- name: APP_VERSION
value: "v3"
strategy:
canary:
canaryService: rollout-canary
stableService: rollout-stable
steps:
- setWeight: 20
- pause: {}
Once the rollout have progressed, we can see the services being updated with the pod template hash to point to the right set of Pods:
$ kubectl get svc rollout-canary -o yaml
apiVersion: v1
kind: Service
metadata:
(...)
name: rollout-canary
spec:
clusterIP: 10.96.51.65
clusterIPs:
- 10.96.51.65
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
selector:
app: canary-rollout
rollouts-pod-template-hash: 7df7c59c9b
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
Other features
While this post covered canary deployments, there are several additional features worth exploring:
- Advanced Traffic Routing: Leveraging ingress controllers and service meshes to dynamically shift traffic based on custom rules and real-time metrics.
- Progressive Experimentation: Using analysis templates and metrics to automatically validate new versions before promoting them.
- Experiments: Running A/B tests and other experiments to compare different versions of an application.
- Automated Analysis: Integrating metrics-based analysis (prometheus, CloudWatch...) to make rollout decisions based on real-time performance data.
You can also checkout blue-green deployments with Argo Rollouts.
Posted on 18/03/2025