Configure secrets for the GitLab chart

GitLab requires a variety of secrets to operate:

GitLab Components:

  • Registry authentication certificates
  • SSH Host Keys and Certificates for GitLab Shell
  • Passwords for individual GitLab services
  • TLS certificate for GitLab Pages

Optional External Services:

  • SMTP server
  • LDAP
  • OmniAuth
  • IMAP for incoming emails (via mail_room service)
  • IMAP for Service Desk emails (via mail_room service)
  • Microsoft Graph with OAuth2 for incoming emails (via mail_room service)
  • Microsoft Graph with OAuth2 for Service Desk email (via mail_room service)
  • Microsoft Graph with OAuth2 for outgoing emails
  • S/MIME certificate
  • Smartcard authentication
  • OAuth integration

Any secret not provided manually will be automatically generated with a random value. Automatic generation of HTTPS certificates is provided by Let’s Encrypt.

To utilize autogenerated secrets continue to next steps.

To specify your own secrets, proceed to manual secret creation.

Manual secret creation (optional)

Use gitlab as the release name if you followed previous steps in this documentation.

Registry authentication certificates

Communication between GitLab and Registry happens behind an Ingress so it is sufficient in most cases to use self-signed certificates for this communication. If this traffic is exposed over a network, you should generate publicly valid certificates.

In the example below, we assume that we require self-signed certificates.

Generate a certificate-key pair:

mkdir -p certs
openssl req -new -newkey rsa:4096 -subj "/CN=gitlab-issuer" -nodes -x509 -keyout certs/registry-example-com.key -out certs/registry-example-com.crt

Create a secret containing these certificates. We will create registry-auth.key and registry-auth.crt keys inside the <name>-registry-secret secret. Replace <name> with the name of the release.

kubectl create secret generic <name>-registry-secret --from-file=registry-auth.key=certs/registry-example-com.key --from-file=registry-auth.crt=certs/registry-example-com.crt

This secret is referenced by the global.registry.certificate.secret setting.

Registry sensitive notification headers

Check documentation regarding configuring Registry notifications for more details.

The secret content should be a list of items, even if it contains a single item. If the content is just a string, the charts WILL NOT convert it to a list as needed.

Consider the example where registry-authorization-header secret with value RandomFooBar is created.

kubectl create secret generic registry-authorization-header --from-literal=value="[RandomFooBar]"

By default, the key used within the secret is “value”. However, users can use a different key, but must ensure that it’s specified as key under the header map item.

SSH Host Keys

Generate the OpenSSH certificate-key pairs:

mkdir -p hostKeys
ssh-keygen -t rsa  -f hostKeys/ssh_host_rsa_key -N ""
ssh-keygen -t dsa  -f hostKeys/ssh_host_dsa_key -N ""
ssh-keygen -t ecdsa  -f hostKeys/ssh_host_ecdsa_key -N ""
ssh-keygen -t ed25519  -f hostKeys/ssh_host_ed25519_key -N ""

Create the secret containing these certificates. Replace <name> with the name of the release.

kubectl create secret generic <name>-gitlab-shell-host-keys --from-file hostKeys

This secret is referenced by the global.shell.hostKeys.secret setting.

If this secret is rotated, all SSH clients will see hostname mismatch errors.

Initial Enterprise license

caution
This method will only add a license at the time of installation. Use the Admin Area in the web user interface to renew or upgrade licenses.

Create a Kubernetes secret for storing the Enterprise license for the GitLab instance. Replace <name> with the name of the release.

kubectl create secret generic <name>-gitlab-license --from-file=license=/tmp/license.gitlab

Then use --set global.gitlab.license.secret=<name>-gitlab-license to inject the license into your configuration.

You can also use the global.gitlab.license.key option to change the default license key pointing to the license in the license secret.

Initial root password

Create a Kubernetes secret for storing the initial root password. The password should be at least 6 characters long. Replace <name> with the name of the release.

kubectl create secret generic <name>-gitlab-initial-root-password --from-literal=password=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32)

Redis password

Generate a random 64 character alpha-numeric password for Redis. Replace <name> with the name of the release.

kubectl create secret generic <name>-redis-secret --from-literal=secret=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 64)

If deploying with an already existing Redis cluster, please use the password for accessing the Redis cluster that has been base64 encoded instead of a randomly generated one.

This secret is referenced by the global.redis.auth.secret setting.

GitLab Shell secret

Generate a random 64 character alpha-numeric secret for GitLab Shell. Replace <name> with the name of the release.

kubectl create secret generic <name>-gitlab-shell-secret --from-literal=secret=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 64)

This secret is referenced by the global.shell.authToken.secret setting.

Gitaly secret

Generate a random 64 character alpha-numeric token for Gitaly. Replace <name> with the name of the release.

kubectl create secret generic <name>-gitaly-secret --from-literal=token=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 64)

This secret is referenced by the global.gitaly.authToken.secret setting.

Praefect secret

Generate a random 64 character alpha-numeric token for Praefect. Replace <name> with the name of the release:

kubectl create secret generic <name>-praefect-secret --from-literal=token=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 64)

This secret is referenced by the global.praefect.authToken.secret setting.

GitLab Rails secret

Replace <name> with the name of the release.

cat << EOF > secrets.yml
production:
  secret_key_base: $(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 128)
  otp_key_base: $(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 128)
  db_key_base: $(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 128)
  encrypted_settings_key_base: $(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 128)
  openid_connect_signing_key: |
$(openssl genrsa 2048 | awk '{print "    " $0}')
  ci_jwt_signing_key: |
$(openssl genrsa 2048 | awk '{print "    " $0}')
EOF

kubectl create secret generic <name>-rails-secret --from-file=secrets.yml

This secret is referenced by the global.railsSecrets.secret setting.

It is not recommended to rotate this secret as it contains the database encryption keys. If the secret is rotated, the result will be the same behavior exhibited when the secrets file is lost.

note
The encrypted_settings_key_base was added in GitLab 13.7, and will be required for GitLab 14.0.

GitLab Workhorse secret

Generate the workhorse secret. This must have a length of 32 characters and base64-encoded. Replace <name> with the name of the release.

kubectl create secret generic <name>-gitlab-workhorse-secret --from-literal=shared_secret=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32 | base64)

This secret is referenced by the global.workhorse.secret setting.

GitLab Runner secret

Replace <name> with the name of the release.

kubectl create secret generic <name>-gitlab-runner-secret --from-literal=runner-registration-token=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 64)

This secret is referenced by the gitlab-runner.runners.secret setting.

GitLab KAS secret

GitLab Rails requires that a secret for KAS is present, even if one deploys this chart without installing the KAS sub-chart. Still, one can create this secret manually by following the below procedure or leave it to the chart to auto-generate the secret.

Replace <name> with the name of the release.

kubectl create secret generic <name>-gitlab-kas-secret --from-literal=kas_shared_secret=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32 | base64)

This secret is referenced by the global.appConfig.gitlab_kas.key setting.

GitLab KAS API secret

You can leave it to the chart to auto-generate the secret, or you can create this secret manually (replace <name> with the name of the release):

kubectl create secret generic <name>-kas-private-api --from-literal=kas_private_api_secret=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32 | base64)

This secret is referenced by the gitlab.kas.privateApi.secret setting.

GitLab Suggested Reviewers secret

note
The Suggested Reviewers secret is created automatically and only used on GitLab SaaS. This secret is not needed on self-managed GitLab instances.

GitLab Rails requires that a secret for Suggested Reviewers is present. You can leave it to the chart to auto-generate the secret, or you can create this secret manually (replace <name> with the name of the release):

kubectl create secret generic <name>-gitlab-suggested-reviewers --from-literal=suggested_reviewers_secret=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32 | base64)

This secret is referenced by the global.appConfig.suggested_reviewers.secret setting.

MinIO secret

Generate a set of random 20 & 64 character alpha-numeric keys for MinIO. Replace <name> with the name of the release.

kubectl create secret generic <name>-minio-secret --from-literal=accesskey=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 20) --from-literal=secretkey=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 64)

This secret is referenced by the global.minio.credentials.secret setting.

PostgreSQL password

Generate a random 64 character alpha-numeric password. Replace <name> with the name of the release.

kubectl create secret generic <name>-postgresql-password \
    --from-literal=postgresql-password=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 64) \
    --from-literal=postgresql-postgres-password=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 64)

This secret is referenced by the global.psql.password.secret setting.

Changing the PostgreSQL password for the bundled PostgreSQL subchart

caution
The default Helm chart configuration is not intended for production, which includes the bundled PostgreSQL subchart.

The bundled PostgreSQL subchart only configures the database with the passwords from the secret when the database is initially created. Additional steps need to be taken to change the passwords in an existing database.

Please note this operation will be disruptive to users while the change is being made.

To rotate the PostgreSQL secret:

  1. Complete the general rotating secrets instructions for the PostgreSQL secret.

  2. Exec into the PostgreSQL pod and update the passwords in the database:

     # Exec into the PostgreSQL pod
 kubectl exec -it <name>-postgresql-0 -- sh

 # Inside the pod, update the passwords in the database
 sed -i 's/^\(local .*\)md5$/\1trust/' /opt/bitnami/postgresql/conf/pg_hba.conf
 pg_ctl reload ; sleep 1
 echo "ALTER USER postgres WITH PASSWORD '$(cat $POSTGRES_POSTGRES_PASSWORD_FILE)' ; ALTER USER gitlab WITH PASSWORD '$(cat $POSTGRES_PASSWORD_FILE)'" | psql -U postgres -d gitlabhq_production -f -
 sed -i 's/^\(local .*\)trust$/\1md5/' /opt/bitnami/postgresql/conf/pg_hba.conf
 pg_ctl reload
  3. Delete the gitlab-exporter, postgresql, toolbox, sidekiq and webservice pods using the kubectl delete pod command so the new pods are loaded with the new secret and allow them to connect to the database.

GitLab Pages secret

Generate the GitLab Pages secret. This must have a length of 32 characters and base64-encoded. Replace <name> with the name of the release.

kubectl create secret generic <name>-gitlab-pages-secret --from-literal=shared_secret=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32 | base64)

This secret is referenced by the global.pages.apiSecret.secret setting.

Registry HTTP secret

Generate a random 64 character alpha-numeric key shared by all registry pods. Replace <name> with the name of the release.

kubectl create secret generic <name>-registry-httpsecret --from-literal=secret=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 64 | base64)

This secret is referenced by the global.registry.httpSecret.secret setting.

Registry notification secret

Generate a random 32 character alpha-numeric key shared by all registry pods, and the GitLab webservice pods. Replace <name> with the name of the release.

kubectl create secret generic <name>-registry-notification --from-literal=secret=[\"$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32)\"]

This secret is referenced by the global.registry.notificationSecret.secret setting.

Praefect DB password

Generate a random 64 character alpha-numeric password. Replace <name> with the name of the release:

kubectl create secret generic <name>-praefect-dbsecret \
    --from-literal=secret=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 64) \

This secret is referenced by the global.praefect.dbSecret setting.

External services

Some charts have further secrets to enable functionality that can not be automatically generated.

OmniAuth

In order to enable the use of OmniAuth Providers with the deployed GitLab, please follow the instructions in the Globals chart

LDAP Password

If you need password authentication to connect with your LDAP server, you must store the password in a Kubernetes secret.

kubectl create secret generic ldap-main-password --from-literal=password=yourpasswordhere

Then use --set global.appConfig.ldap.servers.main.password.secret=ldap-main-password to inject the password into your configuration.

note
Use the Secret name, not the actual password when configuring the Helm property.

SMTP password

If you are using an SMTP server that requires authentication, store the password in a Kubernetes secret.

kubectl create secret generic smtp-password --from-literal=password=yourpasswordhere

Then use --set global.smtp.password.secret=smtp-password in your Helm command.

note
Use the Secret name, not the actual password when configuring the Helm property.

IMAP password for incoming emails

To let GitLab have access to incoming emails store the password of the IMAP account in a Kubernetes secret.

kubectl create secret generic incoming-email-password --from-literal=password=yourpasswordhere

Then use --set global.appConfig.incomingEmail.password.secret=incoming-email-password in your Helm command along with other required settings as specified in the docs.

note
Use the Secret name, not the actual password when configuring the Helm property.

IMAP password for Service Desk emails

To let GitLab have access to service_desk emails store the password of the IMAP account in a Kubernetes secret.

kubectl create secret generic service-desk-email-password --from-literal=password=yourpasswordhere

Then use --set global.appConfig.serviceDeskEmail.password.secret=service-desk-email-password in your Helm command along with other required settings as specified in the docs.

note
Use the Secret name, not the actual password when configuring the Helm property.

GitLab incoming email auth token

When incoming email is configured to use webhook delivery method, there should be a shared secret between mail_room service and webservice. This must have a length of 32 characters and base64-encoded. Replace <name> with the name of the release.

kubectl create secret generic <name>-incoming-email-auth-token --from-literal=authToken=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32 | base64)

This secret is referenced by the global.incomingEmail.authToken setting.

GitLab Service Desk email auth token

When Service Desk email is configured to use webhook delivery method, there should be a shared secret between mail_room service and webservice. This must have a length of 32 characters and base64-encoded. Replace <name> with the name of the release.

kubectl create secret generic <name>-service-desk-email-auth-token --from-literal=authToken=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32 | base64)

This secret is referenced by the global.serviceDeskEmail.authToken setting.

Zoekt basic auth password

You can leave it to the chart to auto-generate the secret, or you can create this secret manually (replace <name> with the name of the release):

password=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32 | base64)
kubectl create secret generic <name>-zoekt-basicauth --from-literal=gitlab_username=gitlab --from-literal=gitlab_password="$password"

This secret is referenced by the gitlab.zoekt.gateway.basicAuth.secretName setting.

Microsoft Graph client secret for incoming emails

To let GitLab have access to incoming emails store the password of the IMAP account in a Kubernetes secret:

kubectl create secret generic incoming-email-client-secret --from-literal=secret=your-secret-here

Then, use --set global.appConfig.incomingEmail.clientSecret.secret=incoming-email-client-secret in your Helm command along with other required settings as specified in the docs.

note
Use the Secret name, not the actual password when configuring the Helm property.

Microsoft Graph client secret for Service Desk emails

To let GitLab have access to service_desk emails store the password of the IMAP account in a Kubernetes secret:

kubectl create secret generic service-desk-email-client-secret --from-literal=secret=your-secret-here

Then, use --set global.appConfig.serviceDeskEmail.clientSecret.secret=service-desk-email-client-secret in your Helm command along with other required settings as specified in the docs.

note
Use the Secret name, not the actual password when configuring the Helm property.

Microsoft Graph client secret for outgoing emails

Store the password in a Kubernetes secret:

kubectl create secret generic microsoft-graph-mailer-client-secret --from-literal=secret=your-secret-here

Then, use --set global.appConfig.microsoft_graph_mailer.client_secret.secret=microsoft-graph-mailer-client-secret in your Helm command.

note
Use the Secret name, not the actual password when configuring the Helm property.

S/MIME Certificate

Outgoing email messages can be digitally signed using the S/MIME standard. The S/MIME certificate needs to be stored in a Kubernetes secret as a TLS type secret.

kubectl create secret tls smime-certificate --key=file.key --cert file.crt

If there is an existing secret as a opaque type, then the global.email.smime.keyName and global.email.smime.certName values will need to be adjusted for the specific secret.

S/MIME settings can be set through the values.yaml file or on the command line. Use --set global.email.smime.enabled=true to enable S/MIME and --set global.email.smime.secretName=smime-certificate to specify the secret that contains the S/MIME certificate.

Smartcard Authentication

Smartcard authentication uses a custom Certificate Authority (CA) to sign client certificates. The certificate of this custom CA needs to be injected to the Webservice pod for it to verify whether a client certificate is valid or not. This is provided as a k8s secret.

kubectl create secret generic <secret name> --from-file=ca.crt=<path to CA certificate>

The key name inside the secret where the certificate is stored MUST BE ca.crt.

OAuth integration

For configuring OAuth integration of various services like GitLab Pages, secrets containing OAuth credentials are required. The secret should contain an App ID (by default, stored under the appid key), and an App Secret (by default, stored under the appsecret key), both of which are recommended to be alphanumeric strings, at least 64 characters long.

kubectl create secret generic oauth-gitlab-pages-secret --from-literal=appid=<app id> --from-literal=appsecret=<app secret>

This secret can be specified using the global.oauth.<service name>.secret setting. If keys other than appid and appsecret are used, they can be specified using global.oauth.<service name>.appIdKey and global.oauth.<service name>.appSecretKey settings.

Next steps

Once all secrets have been generated and stored, you can proceed deploying GitLab.

Rotating secrets

The secrets can be rotated if required for security purposes.

  1. Back up your current secrets.
  2. For your convenience, create new secrets that are suffixed with -v2 (for example gitlab-shell-host-keys-v2) by following the manual secret creation steps for each secret you wish to rotate.
  3. Update the secret keys in your values.yaml file to point to the new secret names. Most secret names are documented under each respective secret in the manual secret creation section.
  4. Upgrade the GitLab Chart release with the updated values.yaml file.
  5. If you are rotating the PostgreSQL secret, there are additional steps to complete the rotation.
  6. Confirm that GitLab is working as expected. If it is, it should be safe to delete the old secrets.