Upgrade the bundled PostgreSQL version

note
These steps are if you are using the bundled PostgreSQL chart (postgresql.install is not false), and not for external PostgreSQL setups.

Changing to a new major version of PostgreSQL using the bundle PostgreSQL chart is done via a backup on the existing database, then restoring to the new database.

note
As part of the 7.0.0 release of this chart, we upgraded the default PostgreSQL version from 12.7.0 to 14.8.0. This is done by upgrading PostgreSQL chart version from 8.9.4 to 12.5.2.

This is NOT a drop in replacement. Manual steps need to be performed to upgrade the database. The steps have been documented in the upgrade steps.

note
As part of the 5.0.0 release of this chart, we upgraded the bundled PostgreSQL version from 11.9.0 to 12.7.0. This is not a drop in replacement. Manual steps need to be performed to upgrade the database. The steps have been documented in the upgrade steps.
note
As part of the 4.0.0 release of this chart, we upgraded the bundled PostgreSQL chart from 7.7.0 to 8.9.4. This is not a drop in replacement. Manual steps need to be performed to upgrade the database. The steps have been documented in the upgrade steps.

Steps for upgrading the bundled PostgreSQL

note
Starting from 7.0.0, GitLab chart not longer mounts PostgreSQL credentials as files inside of the PostgreSQL instance. This is done by setting postgresql.auth.usePasswordFiles to false. This means that database crededntials are passed as environment variables instead of password files, only for this component.

This is due to an issue in upstream PostgreSQL chart. If you do not want to use environment variables for PostgreSQL passwords and prefer to use files you need to follow the instructions for manual editing the existing PostgreSQL passwords Secret and enabling password files for PostgreSQL chart before preforming the following steps.

Prepare the existing database

Note the following:

  • If you are not using the bundled PostgreSQL chart (postgresql.install is false), you do not need to follow these steps.
  • If you have multiple charts installed in the same namespace. It may be necessary to pass the Helm release name to the database-upgrade script as well. Replace bash -s STAGE with bash -s -- -r RELEASE STAGE in the example commands provided later.
  • If you installed a chart to a namespace other than your kubectl context’s default, you must pass the namespace to the database-upgrade script. Replace bash -s STAGE with bash -s -- -n NAMESPACE STAGE in the example commands provided later. This option can be used along with -r RELEASE. You can set the context’s default namespace by running kubectl config set-context --current --namespace=NAMESPACE, or using kubens from kubectx.

The pre stage will create a backup of your database using the backup-utility script in the Toolbox, which gets saved to the configured s3 bucket (MinIO by default):

# GITLAB_RELEASE should be the version of the chart you are installing, starting with 'v': v6.0.0
curl -s "https://gitlab.com/gitlab-org/charts/gitlab/-/raw/${GITLAB_RELEASE}/scripts/database-upgrade" | bash -s pre

Delete existing PostgreSQL data

note
Since the PostgreSQL data format has changed, upgrading requires removing the existing PostgreSQL StatefulSet before upgrading the release. The StatefulSet will be recreated in the next step.
caution
Ensure that you have created a database backup in the previous step. Without a backup, GitLab data will be lost.
kubectl delete statefulset RELEASE-NAME-postgresql
kubectl delete pvc data-RELEASE_NAME-postgresql-0

Upgrade GitLab

Upgrade GitLab following our standard procedure, with the following additions of:

Disable migrations using the following flag on your upgrade command:

  1. --set gitlab.migrations.enabled=false

We will perform the migrations for the Database in a later step for the bundled PostgreSQL.

Restore the Database

Note the following:

  • You’ll need to be using Bash 4.0 or above to run the script successfully as it requires the use of bash associative arrays.
  1. Wait for the upgrade to complete for the Toolbox pod. RELEASE_NAME should be the name of the GitLab release from helm list

    kubectl rollout status -w deployment/RELEASE_NAME-toolbox
    
  2. After the Toolbox pod is deployed successfully, run the post steps:

    # GITLAB_RELEASE should be the version of the chart you are installing, starting with 'v': v6.0.0
    curl -s "https://gitlab.com/gitlab-org/charts/gitlab/-/raw/${GITLAB_RELEASE}/scripts/database-upgrade" | bash -s post
    

    This step will do the following:

    1. Set replicas to 0 for the webservice, sidekiq, and gitlab-exporter deployments. This will prevent any other application from altering the database while the backup is being restored.
    2. Restore the database from the backup created in the pre stage.
    3. Run database migrations for the new version.
    4. Resume the deployments from the first step.

Troubleshooting database upgrade process

  • If you see any failure during the upgrade, it may be useful to check the description of gitlab-upgrade-check pod for details:

    kubectl get pods -lrelease=RELEASE,app=gitlab
    kubectl describe pod <gitlab-upgrade-check-pod-full-name>
    

Edit the existing PostgreSQL passwords Secret

note
This is only for 7.0.0 upgrade, and only when you want enforce the use password files inside of the PostgreSQL service containers.

The new version of PostgreSQL chart uses different keys to reference passwords in a Secrets. Instead of postgresql-password and postgresql-postgres-password it now uses password and postgres-password. These keys must be changed in RELEASE-postgresql-password Secret WITHOUT changing their values.

This Secret is generated by GitLab chart for the first time and does not change during or after upgrades. Therefore you need to edit the Secret and change the keys.

After editing the secret you MUST set postgresql.auth.usePasswordFiles to true in Helm upgrade values. The default is false.

The follwoing script can help you to patch the secret:

  1. First create a backup of the existing Secret. The following command copies it into a new Secret with -backup name suffix:

    kubectl get secrets ${RELEASE}-postgresql-password -o yaml | sed 's/name: \(.*\)$/name: \1-backup/' | kubectl apply -f -
    
  2. Ensure that the patch looks correct:

    kubectl get secret ${RELEASE}-postgresql-password \
      -o go-template='{"data":{"password":"{{index .data "postgresql-password"}}","postgres-password":"{{index .data "postgresql-postgres-password"}}","postgresql-password":null,"postgresql-postgres-password":null}}'
    
  3. Then apply it:

    kubectl patch secret ${RELEASE}-postgresql-password --patch "$(
      kubectl get secret ${RELEASE}-postgresql-password \
        -o go-template='{"data":{"password":"{{index .data "postgresql-password"}}","postgres-password":"{{index .data "postgresql-postgres-password"}}","postgresql-password":null,"postgresql-postgres-password":null}}')"