Container Registry for a secondary site

You can set up a Container Registry on your secondary Geo site that mirrors the one on the primary Geo site.

note
The Container Registry replication is used only for disaster recovery purposes. We do not recommend pulling the Container Registry data from the secondary. For a feature proposal to implement it in the future, see Geo: Accelerate container images by serving read request from secondary site for details.

Supported container registries

Geo supports the following types of container registries:

Supported image formats

The following container image formats are support by Geo:

In addition, Geo also supports BuildKit cache images.

Supported storage

Docker

For more information on supported registry storage drivers see Docker registry storage drivers

Read the Load balancing considerations when deploying the Registry, and how to set up the storage driver for the GitLab integrated Container Registry.

Registries that support OCI artifacts

The following registries support OCI artifacts:

  • CNCF Distribution - local/offline verification
  • Azure Container Registry (ACR)
  • Amazon Elastic Container Registry (ECR)
  • Google Artifact Registry (GAR)
  • GitHub Packages container registry (GHCR)
  • Bundle Bar

For more information, see the OCI Distribution Specification.

Configure Container Registry replication

You can enable a storage-agnostic replication so it can be used for cloud or local storage. Whenever a new image is pushed to the primary site, each secondary site pulls it to its own container repository.

To configure Container Registry replication:

  1. Configure the primary site.
  2. Configure the secondary site.
  3. Verify Container Registry replication.

Configure primary site

Make sure that you have Container Registry set up and working on the primary site before following the next steps.

To be able to replicate new container images, the Container Registry must send notification events to the primary site for every push. The token shared between the Container Registry and the web nodes on the primary is used to make communication more secure.

  1. SSH into your GitLab primary server and login as root (for GitLab HA, you only need a Registry node):

    sudo -i

  2. Edit /etc/gitlab/gitlab.rb:

    registry['notifications'] = [
  {
    'name' => 'geo_event',
    'url' => 'https://<example.com>/api/v4/container_registry_event/events',
    'timeout' => '500ms',
    'threshold' => 5,
    'backoff' => '1s',
    'headers' => {
      'Authorization' => ['<replace_with_a_secret_token>']
    }
  }
]
    note
    Replace <example.com> with the external_url defined in your primary site’s /etc/gitlab/gitlab.rb file, and replace <replace_with_a_secret_token> with a case sensitive alphanumeric string that starts with a letter. You can generate one with < /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo
    note
    If you use an external Registry (not the one integrated with GitLab), you only need to specify the notification secret (registry['notification_secret']) in the /etc/gitlab/gitlab.rb file.

  3. For GitLab HA only. Edit /etc/gitlab/gitlab.rb on every web node:

    registry['notification_secret'] = '<replace_with_a_secret_token_generated_above>'

  4. Reconfigure each node you just updated:

    gitlab-ctl reconfigure

Configure secondary site

Make sure you have Container Registry set up and working on the secondary site before following the next steps.

The following steps should be done on each secondary site you’re expecting to see the container images replicated.

Because we need to allow the secondary site to communicate securely with the primary site Container Registry, we need to have a single key pair for all the sites. The secondary site uses this key to generate a short-lived JWT that is pull-only-capable to access the primary site Container Registry.

For each application and Sidekiq node on the secondary site:

  1. SSH into the node and login as the root user:

    sudo -i

  2. Copy /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key from the primary to the node.

  3. Edit /etc/gitlab/gitlab.rb and add:

    gitlab_rails['geo_registry_replication_enabled'] = true

# Primary registry's hostname and port, it will be used by
# the secondary node to directly communicate to primary registry
gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/'

  4. Reconfigure the node for the change to take effect:

    gitlab-ctl reconfigure

Verify replication

To verify Container Registry replication is working, on the secondary site:

  1. On the left sidebar, select Search or go to.
  2. Select Admin Area.
  3. On the left sidebar, select Geo > Nodes. The initial replication, or “backfill”, is probably still in progress.

You can monitor the synchronization process on each Geo site from the primary site’s Geo Nodes dashboard in your browser.