Using the Praefect chart (alpha)

caution
The Praefect chart is still under development. The alpha version is not yet suitable for production use. Upgrades may require significant manual intervention. See our Praefect GA release Epic for more information.

The Praefect chart is used to manage a Gitaly cluster inside a GitLab installment deployed with the Helm charts.

Known limitations and issues

  1. The database has to be manually created.
  2. The cluster size is fixed: Gitaly Cluster does not currently support autoscaling.
  3. Using a Praefect instance in the cluster to manage Gitaly instances outside the cluster is not supported.
  4. Upgrades to version 4.8 of the chart (GitLab 13.8) will encounter an issue that makes it appear that repository data is lost. Data is not lost, but requires manual intervention.

Requirements

This chart consumes the Gitaly chart. Settings from global.gitaly are used to configure the instances created by this chart. Documentation of these settings can be found in Gitaly chart documentation.

Important: global.gitaly.tls is independent of global.praefect.tls. They are configured separately.

By default, this chart will create 3 Gitaly Replicas.

Configuration

The chart is disabled by default. To enable it as part of a chart deploy set global.praefect.enabled=true.

Replicas

The default number of replicas to deploy is 3. This can be changed by setting global.praefect.virtualStorages[].gitalyReplicas with the desired number of replicas. For example:

global:
  praefect:
    enabled: true
    virtualStorages:
    - name: default
      gitalyReplicas: 4
      maxUnavailable: 1

Multiple virtual storages

Multiple virtual storages can be configured (see Gitaly Cluster documentation). For example:

global:
  praefect:
    enabled: true
    virtualStorages:
    - name: default
      gitalyReplicas: 4
      maxUnavailable: 1
    - name: vs2
      gitalyReplicas: 5
      maxUnavailable: 2

This will create two sets of resources for Gitaly. This includes two Gitaly StatefulSets (one per virtual storage).

Administrators can then configure where new repositories are stored.

Persistence

It is possible to provide persistence configuration per virtual storage.

global:
  praefect:
    enabled: true
    virtualStorages:
    - name: default
      gitalyReplicas: 4
      maxUnavailable: 1
      persistence:
        enabled: true
        size: 50Gi
        accessMode: ReadWriteOnce
        storageClass: storageclass1
    - name: vs2
      gitalyReplicas: 5
      maxUnavailable: 2
      persistence:
        enabled: true
        size: 100Gi
        accessMode: ReadWriteOnce
        storageClass: storageclass2

defaultReplicationFactor

defaultReplicationFactor can be configured on each virtual storages. (see configure replication-factor documentation).

global:
  praefect:
    enabled: true
    virtualStorages:
    - name: default
      gitalyReplicas: 5
      maxUnavailable: 2
      defaultReplicationFactor: 3
    - name: secondary
      gitalyReplicas: 4
      maxUnavailable: 1
      defaultReplicationFactor: 2

Migrating to Praefect

note
Group-level wikis cannot be moved using the API at this time.

When migrating from standalone Gitaly instances to a Praefect setup, global.praefect.replaceInternalGitaly can be set to false. This ensures that the existing Gitaly instances are preserved while the new Praefect-managed Gitaly instances are created.

global:
  praefect:
    enabled: true
    replaceInternalGitaly: false
    virtualStorages:
    - name: virtualStorage2
      gitalyReplicas: 5
      maxUnavailable: 2
note
When migrating to Praefect, none of Praefect’s virtual storages can be named default. This is because there must be at least one storage named default at all times, therefore the name is already taken by the non-Praefect configuration.

The instructions to migrate to Gitaly Cluster can then be followed to move data from the default storage to virtualStorage2. If additional storages were defined under global.gitaly.internal.names, be sure to migrate repositories from those storages as well.

After the repositories have been migrated to virtualStorage2, replaceInternalGitaly can be set back to true if a storage named default is added in the Praefect configuration.

global:
  praefect:
    enabled: true
    replaceInternalGitaly: true
    virtualStorages:
    - name: default
      gitalyReplicas: 4
      maxUnavailable: 1
    - name: virtualStorage2
      gitalyReplicas: 5
      maxUnavailable: 2

The instructions to migrate to Gitaly Cluster can be followed again to move data from virtualStorage2 to the newly-added default storage if desired.

Finally, see the repository storage paths documentation to configure where new repositories are stored.

Creating the database

Praefect uses its own database to track its state. This has to be manually created in order for Praefect to be functional.

note
These instructions assume you are using the bundled PostgreSQL server. If you are using your own server, there will be some variation in how you connect.
  1. Log into your database instance:

    kubectl exec -it $(kubectl get pods -l app=postgresql -o custom-columns=NAME:.metadata.name --no-headers) -- bash
    
    PGPASSWORD=$(cat $POSTGRES_POSTGRES_PASSWORD_FILE) psql -U postgres -d template1
    
  2. Create the database user:

    CREATE ROLE praefect WITH LOGIN;
    
  3. Set the database user password.

    By default, the shared-secrets Job will generate a secret for you.

    1. Fetch the password:

      kubectl get secret RELEASE_NAME-praefect-dbsecret -o jsonpath="{.data.secret}" | base64 --decode
      
    2. Set the password in the psql prompt:

      \password praefect
      
  4. Create the database:

    CREATE DATABASE praefect WITH OWNER praefect;
    

Running Praefect over TLS

Praefect supports communicating with client and Gitaly nodes over TLS. This is controlled by the settings global.praefect.tls.enabled and global.praefect.tls.secretName. To run Praefect over TLS follow these steps:

  1. The Helm chart expects a certificate to be provided for communicating over TLS with Praefect. This certificate should apply to all the Praefect nodes that are present. Hence all hostnames of each of these nodes should be added as a Subject Alternate Name (SAN) to the certificate or alternatively, you can use wildcards.

    To know the hostnames to use, check the file /srv/gitlab/config/gitlab.yml file in the Toolbox Pod and check the various gitaly_address fields specified under repositories.storages key within it.

    kubectl exec -it <Toolbox Pod> -- grep gitaly_address /srv/gitlab/config/gitlab.yml
    
note
A basic script for generating custom signed certificates for internal Praefect Pods can be found in this repository. Users can use or refer that script to generate certificates with proper SAN attributes.
  1. Create a TLS Secret using the certificate created.

    kubectl create secret tls <secret name> --cert=praefect.crt --key=praefect.key
    
  2. Redeploy the Helm chart by passing --set global.praefect.tls.enabled=true.

When running Gitaly over TLS, a secret name must be provided for each virtual storage.

global:
  gitaly:
    tls:
      enabled: true
  praefect:
    enabled: true
    tls:
      enabled: true
      secretName: praefect-tls
    virtualStorages:
    - name: default
      gitalyReplicas: 4
      maxUnavailable: 1
      tlsSecretName: default-tls
    - name: vs2
      gitalyReplicas: 5
      maxUnavailable: 2
      tlsSecretName: vs2-tls

Installation command line options

The table below contains all the possible charts configurations that can be supplied to the helm install command using the --set flags.

ParameterDefaultDescription
common.labels{}Supplemental labels that are applied to all objects created by this chart.
failover.enabledtrueWhether Praefect should perform failover on node failure
failover.readonlyAfterfalseWhether the nodes should be in read-only mode after failover
autoMigratetrueAutomatically run migrations on startup
image.repositoryregistry.gitlab.com/gitlab-org/build/cng/gitalyThe default image repository to use. Praefect is bundled as part of the Gitaly image
podLabels{}Supplemental Pod labels. Will not be used for selectors.
ntpHostpool.ntp.orgConfigure the NTP server Praefect should ask the for the current time.
service.namepraefectThe name of the service to create
service.typeClusterIPThe type of service to create
service.internalPort8075The internal port number that the Praefect pod will be listening on
service.externalPort8075The port number the Praefect service should expose in the cluster
init.resources  
init.image  
extraEnvFrom List of extra environment variables from other data sources to expose
logging.level Log level
logging.formatjsonLog format
logging.sentryDsn Sentry DSN URL - Exceptions from Go server
logging.sentryEnvironment Sentry environment to be used for logging
metrics.enabledtrueIf a metrics endpoint should be made available for scraping
metrics.port9236Metrics endpoint port
metrics.separate_database_metricstrueIf true then metrics scrapes will not perform database queries, setting to false may cause performance problems
metrics.path/metricsMetrics endpoint path
metrics.serviceMonitor.enabledfalseIf a ServiceMonitor should be created to enable Prometheus Operator to manage the metrics scraping, note that enabling this removes the prometheus.io scrape annotations
metrics.serviceMonitor.additionalLabels{}Additional labels to add to the ServiceMonitor
metrics.serviceMonitor.endpointConfig{}Additional endpoint configuration for the ServiceMonitor
securityContext.runAsUser1000 
securityContext.fsGroup1000 
securityContext.fsGroupChangePolicy Policy for changing ownership and permission of the volume (requires Kubernetes 1.23)
serviceLabels{}Supplemental service labels
statefulset.strategy{}Allows one to configure the update strategy utilized by the statefulset