Tutorial: Update HashiCorp Vault configuration to use ID Tokens

This tutorial demonstrates how to convert your existing CI/CD secrets configuration to use ID Tokens.

The CI_JOB_JWT variables are deprecated, but updating to ID tokens requires some important configuration changes to work with Vault. If you have more than a handful of jobs, converting everything at once is a daunting task.

From GitLab 15.9 to 15.11, enable the automatic ID token authentication setting to enable ID Tokens and disable CI_JOB_JWT tokens.

In GitLab 16.0 and later you can use ID tokens without any settings changes. Jobs that use secrets:vault automatically do not have CI_JOB_JWT tokens available, Jobs that don’t use secrets:vault can still use CI_JOB_JWT tokens.

This tutorial will focus on v16 onwards, if you are running a slightly older version you will need to toggle the Limit JSON Web Token (JWT) access setting as appropriate.

There isn’t one standard method to migrate to ID tokens, so this tutorial includes two variations for how to convert your existing CI/CD secrets. Choose the method that is most appropriate for your use case:

  1. Update your Vault configuration:
  2. Update your CI/CD Jobs

Prerequisites

This tutorial assumes you are familiar with GitLab CI/CD and Vault.

To follow along, you must have:

  • An instance running GitLab 15.9 or later, or be on GitLab.com.
  • A Vault server that you are already using.
  • CI/CD jobs retrieving secrets from Vault with CI_JOB_JWT.

In the examples below, replace:

  • vault.example.com with the URL of your Vault server.
  • gitlab.example.com with the URL of your GitLab instance.
  • jwt or jwt_v2 with your auth method names.

Method A: Migrate JWT roles to the new Vault auth method

This method creates a second JWT auth method in parallel to the existing one in use. Afterwards all Vault roles used for the GitLab integration are recreated in this new auth method.

Create a second JWT authentication path in Vault

As part of the transition from CI_JOB_JWT to ID tokens, you must update the bound_issuer in Vault to include https://:

$ vault write auth/jwt/config \
    jwks_url="https://gitlab.example.com/-/jwks" \
    bound_issuer="https://gitlab.example.com"

After you make this change, jobs that use CI_JOB_JWT start to fail.

You can create multiple authentication paths in Vault, which enable you to transition to IT Tokens on a project by job basis without disruption.

  1. Configure a new authentication path with the name jwt_v2, run:

    vault auth enable -path jwt_v2 jwt

    You can choose a different name, but the rest of these examples assume you used jwt_v2, so update the examples as needed.

  2. Configure the new authentication path for your instance:

    $ vault write auth/jwt_v2/config \
    jwks_url="https://gitlab.example.com/-/jwks" \
    bound_issuer="https://gitlab.example.com"

Recreate roles to use the new authentication path

Roles are bound to a specific authentication path so you need to add new roles for each job.

  1. Recreate the role for staging named myproject-staging:

    $ vault write auth/jwt_v2/role/myproject-staging - <<EOF
{
  "role_type": "jwt",
  "policies": ["myproject-staging"],
  "token_explicit_max_ttl": 60,
  "user_claim": "user_email",
  "bound_claims": {
    "project_id": "22",
    "ref": "master",
    "ref_type": "branch"
  }
}
EOF

  2. Recreate the role for production named myproject-production:

    $ vault write auth/jwt_v2/role/myproject-production - <<EOF
{
  "role_type": "jwt",
  "policies": ["myproject-production"],
  "token_explicit_max_ttl": 60,
  "user_claim": "user_email",
  "bound_claims_type": "glob",
  "bound_claims": {
    "project_id": "22",
    "ref_protected": "true",
    "ref_type": "branch",
    "ref": "auto-deploy-*"
  }
}
EOF

You only need to update jwt to jwt_v2 in the vault command, do not change the role_type inside the role.

Method B: Move iss claim to roles for migration window

This method doesn’t require Vault administrators to create a second JWT auth method and recreate all GitLab related roles.

Add bound_issuers claim map to each role

Vault doesn’t allow multiple iss claims on the JWT auth method level, as the bound_issuer directive on this level only accepts a single value. However, multiple claims can be configured on the role level by using the bound_claims map configuration directive.

With this method you can provide Vault with multiple options for the iss claim validation. This supports the https:// prefixed GitLab instance hostname claim that comes with the id_tokens, as well as the old non-prefixed claim.

To add the bound_claims configuration to the required roles, run:

$ vault write auth/jwt/role/myproject-staging - <<EOF
{
  "role_type": "jwt",
  "policies": ["myproject-staging"],
  "token_explicit_max_ttl": 60,
  "user_claim": "user_email",
  "bound_claims": {
    "iss": [
      "https://gitlab.example.com",
      "gitlab.example.com"
    ],
    "project_id": "22",
    "ref": "master",
    "ref_type": "branch"
  }
}
EOF

You do not need to alter any existing role configurations except for the bound_claims section Make sure to add the iss configuration as shown above to ensure Vault accepts the prefixed and non-prefixed iss claim for this role.

You must apply this change to all JWT roles used for the GitLab integration before moving on to the next step.

You can revert the migration of the iss claim validation from the auth method to the roles if desired, after all projects have been migrated and you no longer need parallel support for CI_JOB_JWT and ID tokens.

Remove bound_issuers claim from auth method

After all roles have been updated with the bound_claims.iss claims, you can remove the auth method level configuration for this validation:

$ vault write auth/jwt/config \
    jwks_url="https://gitlab.example.com/-/jwks" \
    bound_issuer=""

Setting the bound_issuer directive to an empty string removes the issuer validation on the auth method level. However, as we have moved this validation to the role level, this configuration is still secure.

Update your CI/CD Jobs

Vault has two different KV Secrets Engines and the version you are using impacts how you define secrets in CI/CD.

Check the Which Version is my Vault KV Mount? article on HashiCorp’s support portal to check your Vault server.

Also, if needed you can review the CI/CD documentation for:

The following examples show how to obtain the staging database password written to the password field in secret/myproject/staging/db.

The value for the VAULT_AUTH_PATH variable depends on the migration method you used:

  • Method A (Migrate JWT roles to the new Vault auth method): Use jwt_v2.
  • Method B (Move iss claim to roles for migration window): Use jwt.

KV Secrets Engine v1

The secrets:vault keyword defaults to v2 of the KV Mount, so you need to explicitly configure the job to use the v1 engine:

job:
  variables:
    VAULT_SERVER_URL: https://vault.example.com
    VAULT_AUTH_PATH: jwt_v2  # or "jwt" if you used method B
    VAULT_AUTH_ROLE: myproject-staging
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.example.com
  secrets:
    PASSWORD:
      vault:
        engine:
          name: kv-v1
          path: secret
        field: password
        path: myproject/staging/db
      file: false

Both VAULT_SERVER_URL and VAULT_AUTH_PATH can be defined as project or group CI/CD variables, if preferred.

We use secrets:file:false because ID tokens place secrets in a file by default, but we need it to work as a regular variable to match the old behavior.

KV Secrets Engine v2

There are two formats you can use for the v2 engine.

Long format:

job:
  variables:
    VAULT_SERVER_URL: https://vault.example.com
    VAULT_AUTH_PATH: jwt_v2  # or "jwt" if you used method B
    VAULT_AUTH_ROLE: myproject-staging
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.example.com
  secrets:
    PASSWORD:
      vault:
        engine:
          name: kv-v2
          path: secret
        field: password
        path: myproject/staging/db
      file: false

This is the same as the example for the v1 engine but secrets:vault:engine:name: is set to kv-v2 to match the engine.

You can also use a short format:

job:
  variables:
    VAULT_SERVER_URL: https://vault.example.com
    VAULT_AUTH_PATH: jwt_v2  # or "jwt" if you used method B
    VAULT_AUTH_ROLE: myproject-staging
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.example.com
  secrets:
      PASSWORD:
        vault: myproject/staging/db/password@secret
        file: false

After you commit the updated CI/CD configuration, your jobs will be fetching secrets with ID Tokens, congratulations!

If you have migrated all projects to fetch secrets with ID Tokens and used method B for the migration, it is now possible to move the iss claim validation back to the auth method configuration if you desire.