Repositories, branches, and CI pipelines

omnibus-gitlab CI pipeline is a moderately complex one with pipelines split between all the mirrors of the project:

  1. Development repository for normal feature development.
  2. Release mirror for building release artifacts.
  3. Security mirror for security development.
  4. QA mirror for developers to run a package build and QA for development purposes.

Branch and tag protection

Protected branches

note
Unless specified otherwise, the listed users/groups have permission to both merge, and push to protected branches.
  1. Development repository
    1. master: Maintainers, Delivery team
    2. *-stable : Delivery team, Release Managers
    3. *-stable-ee : Delivery team, Release Managers
    4. *-auto-deploy-* : Maintainers, delivery group, managers group
  2. Release mirror:
    1. master: Maintainers
    2. *-stable : Maintainers
    3. *-stable-ee : Maintainers
    4. *-auto-deploy-* : Maintainers
  3. Security mirror:
    1. master: GitLab Release Tools Bot, GitLab Bot, Delivery team, Release Managers
    2. *-stable: GitLab Release Tools Bot, GitLab Bot, Delivery team, Release Managers
    3. *-stable-ee: GitLab Release Tools Bot, GitLab Bot, Delivery team, Release Managers
    4. *-auto-deploy-*: GitLab Release Tools Bot, GitLab Bot, Delivery team, Release Managers
  4. QA mirror:
    1. master: Developers (Merge only), Maintainers
note
Developers get access to master branch in QA mirror because that’s required to run a triggered pipeline against the branch. There is an open issue to change this situation.

Protected Tags

note
Unless specified otherwise, the listed users/groups have permission to both merge, and push to protected tags.
  1. Development repository:
    1. * : Maintainers, Delivery team, Release Managers
  2. Release mirror:
    1. *: Maintainers

  3. Security mirror: Nil

  4. QA mirror: Nil

Mirroring between repositories

Most of the development happens in the Development repository and security related changes go to Security mirror. These changes then gets mirrored to Release mirror and QA mirror. The following diagram describes what all gets mirrored between each of these repositories.

graph TD A[Development repository] -->|Protected branches| B[Security mirror] A[Development repository] -->|Protected branches| C[Release mirror] A[Development repository] -->|All branches| D[QA mirror] B[Security mirror] -->|Protected branches| C[Release mirror] B[Security mirror] -->|All branches| D[QA mirror]

Types of pipelines

Since the jobs in omnibus-gitlab are spread across multiple mirrors, the CI config becomes relatively complex when using rules keyword for specifying where/when to run jobs. Hence, a different approach is used for omnibus-gitlab, which marks each necessary pipeline type with a specific variable named PIPELINE_TYPE, and attach jobs to various pipeline types as needed. The different pipeline types are documented in the below table:

Pipeline typeMirror(s) where the pipeline runsRemarks
DEPENDENCY_SCANNING_PIPELINECanonicalCheck for security vulnerabilities in dependencies. Requires DEPENDENCY_SCANNING variable to be set to true.
DEPS_IO_VERSION_BUMP_PIPELINECanonicalOn branch pushes done by deps.io. Requires branches to have deps- prefix.
DEPS_IO_VERSION_CHECK_PIPELINECanonicalRun deps to detect updates. Requires DEPS_PIPELINE variable to be set to true.
LICENSE_PAGE_UPDATE_PIPELINECanonicalUpdate license pages. Requires PAGES_UPDATE variable to be set to true.
CACHE_UPDATE_PIPELINECanonical, QAUpdate the gem cache and package build cache. Requires CACHE_UPDATE variable to be set to true.
DURATION_PLOTTER_PIPELINEQAPackage build to plot duration of builds. Requires DURATION_PLOTTER variable to be set to true.
DOCS_PIPELINECanonical, SecurityRequires branch names to either have docs- as prefix or -docs as suffix.
PROTECTED_TEST_PIPELINECanonical, SecurityOn protected branches and tags. Does not include trigger jobs and other unwanted jobs like danger-review.
GITLAB_BRANCH_TEST_PIPELINECanonical, SecurityOn non-protected branch pushes. Does not include unwanted jobs like danger-review. Includes trigger jobs.
GITLAB_MR_PIPELINECanonical, SecurityOn MRs with branches in Canonical/Security (i.e. MRs from GitLab and team members). Has trigger jobs.
AUTO_DEPLOY_BUILD_PIPELINEReleaseBuilds when auto-deploy tags are pushed.
CE_BRANCH_BUILD_PIPELINEReleaseBuilds when regular branches are pushed.
CE_NIGHTLY_BUILD_PIPELINEReleaseNightly CE builds.
CE_RC_BUILD_PIPELINEReleaseBuilds when CE RC tags are pushed.
CE_TAG_BUILD_PIPELINEReleaseBuilds when stable CE tags are pushed.
EE_BRANCH_BUILD_PIPELINEReleaseBuilds when regular branches are pushed, but forced to be an EE pipeline.
EE_NIGHTLY_BUILD_PIPELINEReleaseNightly EE builds. Requires pipeline to be forced to be an EE one.
EE_RC_BUILD_PIPELINEReleaseBuilds when EE RC tags are pushed.
EE_TAG_BUILD_PIPELINEReleaseBuilds when stable EE tags are pushed.
TRIGGER_CACHE_UPDATE_PIPELINEQAUpdates build cache of triggered QA pipelines. Requires CACHE_UPDATE variable to be set to true.
TRIGGERED_CE_PIPELINEQATriggered e2e:package-and-test build with CE packages and images.
TRIGGERED_EE_PIPELINEQATriggered e2e:package-and-test build with EE packages and images.
FORK_BRANCH_TEST_PIPELINEForksTest suite when run on forks of the project. Does not include trigger jobs and other unwanted jobs like danger-review.
FORK_MR_PIPELINEForksOn MRs from forks of the project. Does not include trigger jobs.

Branch pipelines

omnibus-gitlab doesn’t make use of detached merge request pipelines yet. So, pipelines caused by branch pushes are the common pipelines encountered in this project. These pipelines are used in Development repository, Security mirror, and Release mirror.

Tag pipelines

Tag pushes to Release mirror starts a pipeline with the release jobs. Tag pushes to Development repository and Security mirror behaves as regular branch pushes (except that they don’t have an option to start a e2e:package-and-test pipeline) and run basic stylistic checks and unit tests.

Scheduled pipelines

In Development repository, there are two scheduled pipelines.

  1. Dependencies Update - Pipeline to check for outdated dependencies using the dependency_update job.
  2. Generate license pages - Pipeline to populate License collection webpage with license information from the S3 bucket.

In Release mirror, there are two scheduled pipelines

  1. CE nightly - Pipeline to build nightly packages and Docker images for GitLab CE
  2. EE nightly - Pipeline to build nightly packages and Docker images for GitLab EE

Other mirrors doesn’t have any scheduled pipelines

Auto-deploy pipelines

GitLab uses Auto deploy branches and tags for its release process. These branches are named <MAJOR>-<MINOR>-auto-deploy-<TIMESTAMP> and tags have the format <MAJOR>.<MINOR>.<TIMESTAMP>+<gitlab sha>.<omnibus-gitlab sha>.

Only certain jobs - that are needed to deploy the changes to GitLab environments - will be part of this pipeline.

Triggered pipelines

We use triggered pipelines to run a “e2e:package-and-test” pipeline in the QA mirror. These can be triggered either from a pipeline in the Development repository or GitLab project.

This pipeline is intended to give the developers a package and an image to test their changes in addition to automatically doing a QA run against these artifacts. It also provides an option to do a QA run against an HA instance spun up using these artifacts.

CI Jobs

Development Jobs

danger-review

This job runs a check against the merge request to ensure it satisfies some basic requirements using the Danger tool.

This job is run only on Development repository and Security mirror on branch and tag pipelines.

rubocop

Check the source code files to satisfy certain stylistic requirements.

This job is run only on Development repository and Security mirror on branch and tag pipelines.

docs-lint

Check the docs files to satisfy certain stylistic requirements.

This job is run only on Development repository and Security mirror on branch and tag pipelines.

review-docs-deploy

A manual job that will trigger a docs build in gitlab-docs, which will deploy a review app for the GitLab Docs with changes from the current commit.

This job is run only on Development repository on branch pipelines.

review-docs-cleanup

A manual job to stop the environment created by review-docs-deploy. Automatically ran on merging a merge request.

This job is run only on Development repository on branch pipelines.

<OS_NAME> knapsack

We have tests covering our Chef recipes and libraries using RSpec and Chefspec. They are parallelized with the help of knapsack.

This job pulls in the knapsack reports of previous pipeline runs from the cache and gets them in place for the current spec runs.

These jobs are run only on Development repository and Security mirror on branch and tag pipelines.

<OS_NAME> specs

Jobs that actually run rspec via knapsack. Parallelized to a count of 6 using the parallel keyword

These jobs are run only on Development repository and Security mirror on branch and tag pipelines.

update-knapsack

This job combines the reports from each individual parallel runs of unit tests and prepares a final JSON report. This report is used by the MR widget to display spec status, and is also uploaded to the cache for the next pipeline run to use.

This job is run only on Development repository and Security mirror on branch and tag pipelines.

Trigger:ce-package

This is a manual job which when played triggers a pipeline in the QA mirror to run a package build and QA for development purposes. This can be used by developers to get a package or Docker image for testing, or to run a full QA suite against their MR’s changes.

To debug QA failures, refer to the Investigate QA failures section. This job also generates Allure report, see more information and the demo at Test Report Generation in Omnibus-GitLab.

This job is run only on Development repository and Security mirror on branch pipelines.

Trigger:ee-package

Same as Trigger:ce-package, but builds an EE package.

This job is run only on Development repository and Security mirror on branch pipelines.

fetch-assets

For building packages, we make use of the Rails assets that are already compiled by the GitLab or GitLab-FOSS pipelines. Those pipelines push it as a Docker image, which we pull in here and copy the assets themselves to a predefined location.

This job is run only on Release mirror and QA mirror on branch, tag and triggered pipelines.

Trigger:package

This job builds a single package that will be available as an artifact.

This job is run only on QA mirror on triggered pipelines.

Trigger:gitlab-docker

This job builds a GitLab Docker image using the package that was built by Trigger:package job.

This job is run only on QA mirror on triggered pipelines.

qa-subset-test

This job runs subset of test suite and is automatically triggered when either Trigger:CE-package or Trigger:EE-package manual jobs are triggered manually.

This job triggers a pipeline in the GitLab QA Mirror, passing the GitLab Docker image created by Trigger:gitlab-docker job and the GitLab QA Docker image built by the GitLab Rails pipeline, so that a subset of test as mentioned in this issue will run using these images

This job is run only on QA mirror on triggered pipelines.

qa-remaining-test-manual

This is a manual trigger job which runs the remaining tests that donot run in the qa-subset-test job.

To run this QA jobs in MR pipeline, you need to either trigger Trigger:CE-package or Trigger:EE-package manual jobs.

This job triggers a pipeline in the GitLab QA Mirror, passing the GitLab Docker image created by Trigger:gitlab-docker job and the GitLab QA Docker image built by the GitLab Rails pipeline, so that the entire suite is run using these images.

RAT

This manual job triggers a pipeline in the RAT project passing the URL to the package built by Trigger:package job, which will spin up a PostgreSQL HA instance with that package using GET, and run QA against that instance.

This job is run only on QA mirror on triggered EE pipelines.

<OS_NAME>-branch

These jobs build packages for the specified OS, and pushes the resultant package to an S3 bucket in addition to making them available as artifacts.

This job is run only on Release mirror on branch and nightly pipelines.

NOTE: Raspberry Pi jobs are run only on CE branches while SLES jobs are run only on EE branches.

Docker-branch

This job builds a GitLab Docker image using the package built during the Ubuntu 22.04-branch job. The image is pushed to the GitLab container registry.

This job is run only on Release mirror on branch and nightly pipelines.

QA-branch

This job builds a GitLab QA Docker image from the qa directory in the Rails codebase.

This job is run only on Release mirror on branch and nightly pipelines.

Release jobs

<OS_NAME>

These jobs build packages for the specified OS, and pushes the resultant package to an S3 bucket in addition to making them available as artifacts.

This job is run only on Release mirror on tag pipelines.

note
Raspberry Pi jobs are run only on CE tags while SLES jobs are run only on EE tags.

<OS_NAME>-staging

These jobs upload the packages built in previous job to an internal staging repository in Packagecloud instance.

This job is run only on Release mirror on tag pipelines.

note
Raspberry Pi jobs are run only on CE tags while SLES jobs are run only on EE tags.

<OS_NAME>-release

These jobs pulls the packages from the internal staging repository to a public repository in Packagecloud instance.

This job is run only on Release mirror on tag pipelines.

note
Raspberry Pi jobs are run only on CE tags while SLES jobs are run only on EE tags.

Docker

This job builds a GitLab Docker image using the package built during the Ubuntu 22.04-branch job. The image is pushed to the internal GitLab container registry.

This job is run only on Release mirror on tag pipelines.

Docker-Release

This job pulls the GitLab Docker image from internal GitLab container registry and pushes it to Dockerhub.

This job is run only on Release mirror on tag pipelines.

Docker-QA

This job builds a GitLab QA Docker image from the qa directory in the Rails codebase and pushes it to the internal GitLab container registry.

This job is run only on Release mirror and on tag pipelines.

QA-Tag

This job pulls the GitLab QA Docker image from the internal GitLab container registry and pushes it to Dockerhub.

This job is run only on Release mirror and on tag pipelines.

AWS

This job builds the unlicensed AWS AMIs using the Ubuntu 20.04 package.

This job is run only on Release mirror and on tag pipelines.

AWS-Ultimate

This job builds an AWS AMI with embedded Ultimate licenses using the Ubuntu 20.04 package.

This job is run only on Release mirror and on EE tag pipelines.

AWS-Premium

This job builds an AWS AMI with embedded Premium licenses using the Ubuntu 20.04 package.

This job is run only on Release mirror and on EE tag pipelines.

AWS-CE-Release

This job updates the GitLab Community Edition listing in AWS Marketplace with the current version.

This job is run only on Release mirror and on CE tag pipelines.

AWS-Ultimate-Release

This job updates the GitLab Ultimate listing in AWS Marketplace with the current version.

This job is run only on Release mirror and on EE tag pipelines.

AWS-Premium-Release

This job updates the GitLab Premium listing in AWS Marketplace with the current version.

This job is run only on Release mirror and on EE tag pipelines.

create_omnibus_manifest

This job creates a version-manifest.json file to be used by dependency_scanning job.

This job is run only on Release mirror and on tag and nightly pipelines.

dependency_scanning

This job runs a dependency scan on all the components of the package to check if any of them are vulnerable to any known vulnerabilities.

This job is run only on Release mirror and on tag and nightly pipelines.

RAT-Nightly

This job triggers a pipeline in the RAT project passing the URL to the nightly Ubuntu 22.04 package built in this pipeline, which will spin up a PostgreSQL HA instance with that package using GET, and run QA against that instance.

This job is run only on Release mirror on nightly pipelines.

RAT-Tag

This job triggers a pipeline in the RAT project passing the URL to the Ubuntu 22.04 package built in this pipeline, which will spin up a PostgreSQL HA instance with that package using GET, and run QA against that instance.

This job is run only on Release mirror on EE tag pipelines.

license-upload

This job compiles the license information of all the dependencies from the package and uploads it to the S3 bucket. This is used by pages scheduled job in Development repository to populate the License collection webpage.

Housekeeping Jobs

update-gems-cache and update-trigger-package-cache

Jobs that extends from the .gems-cache and .trigger-package-cache shared cache definitions only pull their cache.

These caches are updated in scheduled pipelines by update-gems-cache and update-trigger-package-cache jobs respectively, when the CACHE_UPDATE exists.

dependency_update

This job makes use of Dependencies.io to automatically check for version updates of various components that we include in our package and open merge request against Development repository if an update is found.

This job is run only on Development repository, on scheduled pipelines when DEPS_PIPELINE variable exists.

dependencies_io_check

This job automatically triggers a e2e:package-and-test pipeline in QA mirror, (similar to Trigger:ce-package job), when a merge request is made by dependency_update job.

This job is run only on Development repository when the branch name starts with deps (this is the format that dependency_update job uses for merge requests)

valdiate_packer_changes

This job checks if the packer configuration files are valid or not.

This job is run only on Development repository and Security mirror, when any of the packer configuration files are modified.

pages

This job is associated with GitLab Pages, and generates a static website containing license information of various components included in the package for each release of GitLab.

This job is run only on Development repository and on scheduled pipelines which does not have the DEPS_PIPELINE variable (to differentiate it from dependency_update runs).