Canary deployments
- Introduced in GitLab 9.1.
- Moved from GitLab Premium to GitLab Free in 13.8.
Canary deployments are a popular continuous deployment strategy, where a small portion of the fleet is updated to the new version of your application.
When embracing continuous delivery, an organization needs to decide what type of deployment strategy to use. One of the most popular strategies is canary deployments, where a small portion of the fleet is updated to the new version first. This subset, the canaries, then serve as the proverbial canary in the coal mine.
If there is a problem with the new version of the application, only a small percentage of users are affected and the change can either be fixed or quickly reverted.
Use cases
Canary deployments can be used when you want to ship features to only a portion of your pods fleet and watch their behavior as a percentage of your user base visits the temporarily deployed feature. If all works well, you can deploy the feature to production knowing that it shouldn’t cause any problems.
Canary deployments are also especially required for backend refactors, performance
improvements, or other changes where the user interface doesn’t change, but you
want to make sure the performance stays the same, or improves. Developers need
to be careful when using canaries with user-facing changes, because by default,
requests from the same user are randomly distributed between canary and
non-canary pods, which could result in confusion or even errors. If needed, you
may want to consider
setting service.spec.sessionAffinity
to ClientIP
in your Kubernetes service definitions,
but that is beyond the scope of this document.
Advanced traffic control with Canary Ingress
Canary deployments can be more strategic with Canary Ingress, which is an advanced traffic routing service that controls incoming HTTP requests between stable and canary deployments based on factors such as weight, sessions, cookies, and others. GitLab uses this service in its Auto Deploy architecture to let users quickly and safely roll out their new deployments.
How to set up a Canary Ingress in a canary deployment
A Canary Ingress is installed by default if your Auto DevOps pipeline uses
v2.0.0+
of auto-deploy-image
.
A Canary Ingress becomes available when you create a new canary deployment and is destroyed when the
canary deployment is promoted to production.
Here’s an example setup flow from scratch:
- Prepare an Auto DevOps-enabled project.
- Set up a Kubernetes Cluster in your project.
- Install NGINX Ingress in your cluster.
- Set up the base domain based on the Ingress Endpoint assigned above.
- Check if
v2.0.0+
ofauto-deploy-image
is used in your Auto DevOps pipelines. If it isn’t, follow the documentation to specify the image version. -
Run a new Auto DevOps pipeline
and make sure that the
production
job succeeds and creates a production environment. - Configure a
canary
deployment job for Auto DevOps pipelines. -
Run a new Auto DevOps pipeline
and make sure that the
canary
job succeeds and creates a canary deployment with Canary Ingress.
Show Canary Ingress deployments on deploy boards (deprecated)
- Introduced in GitLab 13.6.
- Moved from GitLab Premium to GitLab Free in 13.8.
- Deprecated in GitLab 14.5.
To view canary deployments you must properly configure deploy boards:
- Follow the steps to enable deploy boards.
- To track canary deployments you must label your Kubernetes deployments and
pods with
track: canary
. To get started quickly, you can use the Auto Deploy template for canary deployments that GitLab provides.
Depending on the deploy, the label should be either stable
or canary
.
GitLab assumes the track label is stable
if the label is blank or missing.
Any other track label is considered canary
(temporary).
This allows GitLab to discover whether a deployment is stable or canary (temporary).
Once all of the above are set up and the pipeline has run at least once, Go to the environments page under Pipelines > Environments. As the pipeline executes, deploy boards clearly mark canary pods, enabling quick and clear insight into the status of each environment and deployment.
Canary deployments are marked with a yellow dot in the deploy board so that you can quickly notice them.
How to check the current traffic weight on a Canary Ingress (deprecated)
- Visit the deploy board.
-
View the current weights on the right.
How to change the traffic weight on a Canary Ingress (deprecated)
You can change the traffic weight in your environment’s deploy board by using GraphiQL, or by sending requests to the GraphQL API.
To use your deploy board:
- Go to Deployments > Environments for your project.
- Set the new weight with the dropdown list on the right side.
- Confirm your selection.
Here’s an example using GraphiQL:
- Visit GraphiQL Explorer.
-
Execute the
environmentsCanaryIngressUpdate
GraphQL mutation:mutation { environmentsCanaryIngressUpdate(input:{ id: "gid://gitlab/Environment/29", # Your Environment ID. You can get the ID from the URL of the environment page. weight: 45 # The new traffic weight. e.g. If you set `45`, 45% of traffic goes to a canary deployment and 55% of traffic goes to a stable deployment. }) { errors } }
- If the request succeeds, the
errors
response contains an empty array. GitLab sends aPATCH
request to your Kubernetes cluster for updating the weight parameter on a Canary Ingress.