Terraform state administration
Introduced in GitLab 12.10.
GitLab can be used as a backend for Terraform state files. The files are encrypted before being stored. This feature is enabled by default.
The storage location of these files defaults to:
-
/var/opt/gitlab/gitlab-rails/shared/terraform_state
for Linux package installations. -
/home/git/gitlab/shared/terraform_state
for self-compiled installations.
These locations can be configured using the options described below.
Use external object storage configuration for GitLab Helm chart installations.
Disabling Terraform state
You can disable Terraform state across the entire instance. You might want to disable Terraform to reduce disk space, or because your instance doesn’t use Terraform.
When Terraform state administration is disabled:
- On the left sidebar, you cannot select Operate > Terraform states.
-
Any CI/CD jobs that access the Terraform state fail with this error:
Error refreshing state: HTTP remote state endpoint invalid auth
To disable Terraform administration, follow the steps below according to your installation.
Prerequisite:
- You must be an administrator.
For Linux package installations:
-
Edit
/etc/gitlab/gitlab.rb
and add the following line:gitlab_rails['terraform_state_enabled'] = false
-
Save the file and reconfigure GitLab for the changes to take effect.
For self-compiled installations:
-
Edit
/home/git/gitlab/config/gitlab.yml
and add or amend the following lines:terraform_state: enabled: false
-
Save the file and restart GitLab for the changes to take effect.
Using local storage
The default configuration uses local storage. To change the location where Terraform state files are stored locally, follow the steps below.
For Linux package installations:
-
To change the storage path for example to
/mnt/storage/terraform_state
, edit/etc/gitlab/gitlab.rb
and add the following line:gitlab_rails['terraform_state_storage_path'] = "/mnt/storage/terraform_state"
-
Save the file and reconfigure GitLab for the changes to take effect.
For self-compiled installations:
-
To change the storage path for example to
/mnt/storage/terraform_state
, edit/home/git/gitlab/config/gitlab.yml
and add or amend the following lines:terraform_state: enabled: true storage_path: /mnt/storage/terraform_state
-
Save the file and restart GitLab for the changes to take effect.
Using object storage
Instead of storing Terraform state files on disk, we recommend the use of one of the supported object storage options. This configuration relies on valid credentials to be configured already.
Read more about using object storage with GitLab.
Object storage settings
The following settings are:
- Prefixed by
terraform_state_object_store_
on Linux package installations. - Nested under
terraform_state:
and thenobject_store:
on self-compiled installations.
Setting | Description | Default |
---|---|---|
enabled | Enable/disable object storage | false |
remote_directory | The bucket name where Terraform state files are stored | |
connection | Various connection options described below |
Migrate to object storage
Introduced in GitLab 13.9.
To migrate Terraform state files to object storage:
-
For Linux package installations:
gitlab-rake gitlab:terraform_states:migrate
-
For self-compiled installations:
sudo -u git -H bundle exec rake gitlab:terraform_states:migrate RAILS_ENV=production
For GitLab 13.8 and earlier versions, you can use a workaround for the Rake task:
- Open the GitLab Rails console.
-
Run the following commands:
Terraform::StateUploader.alias_method(:upload, :model) Terraform::StateVersion.where(file_store: ::ObjectStorage::Store::LOCAL). find_each(batch_size: 10) do |terraform_state_version| puts "Migrating: #{terraform_state_version.inspect}" terraform_state_version.file.migrate!(::ObjectStorage::Store::REMOTE) end
You can optionally track progress and verify that all Terraform state files migrated successfully using the PostgreSQL console:
-
sudo gitlab-rails dbconsole
for Linux package installations running GitLab 14.1 and earlier. -
sudo gitlab-rails dbconsole --database main
for Linux package installations running GitLab 14.2 and later. -
sudo -u git -H psql -d gitlabhq_production
for self-compiled installations.
Verify objectstg
below (where file_store=2
) has count of all states:
gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM terraform_state_versions;
total | filesystem | objectstg
------+------------+-----------
15 | 0 | 15
Verify that there are no files on disk in the terraform_state
folder:
sudo find /var/opt/gitlab/gitlab-rails/shared/terraform_state -type f | grep -v tmp | wc -l
S3-compatible connection settings
In GitLab 13.2 and later, you should use the consolidated object storage settings. This section describes the earlier configuration format.
See the available connection settings for different providers.
-
Edit
/etc/gitlab/gitlab.rb
and add the following lines; replacing with the values you want:gitlab_rails['terraform_state_object_store_enabled'] = true gitlab_rails['terraform_state_object_store_remote_directory'] = "terraform" gitlab_rails['terraform_state_object_store_connection'] = { 'provider' => 'AWS', 'region' => 'eu-central-1', 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' }
If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs.gitlab_rails['terraform_state_object_store_connection'] = { 'provider' => 'AWS', 'region' => 'eu-central-1', 'use_iam_profile' => true }
- Save the file and reconfigure GitLab for the changes to take effect.
- Migrate any existing local states to the object storage
-
Edit
/home/git/gitlab/config/gitlab.yml
and add or amend the following lines:terraform_state: enabled: true object_store: enabled: true remote_directory: "terraform" # The bucket name connection: provider: AWS # Only AWS supported at the moment aws_access_key_id: AWS_ACCESS_KEY_ID aws_secret_access_key: AWS_SECRET_ACCESS_KEY region: eu-central-1
- Save the file and restart GitLab for the changes to take effect.
- Migrate any existing local states to the object storage
Find a Terraform state file path
Terraform state files are stored in the hashed directory path of the relevant project.
The format of the path is /var/opt/gitlab/gitlab-rails/shared/terraform_state/<path>/<to>/<projectHashDirectory>/<UUID>/0.tfstate
, where UUID is randomly defined.
To find a state file path:
-
Add
get-terraform-path
to your shell:get-terraform-path() { PROJECT_HASH=$(echo -n $1 | openssl dgst -sha256 | sed 's/^.* //') echo "${PROJECT_HASH:0:2}/${PROJECT_HASH:2:2}/${PROJECT_HASH}" }
-
Run
get-terraform-path <project_id>
.$ get-terraform-path 650 20/99/2099a9b5f777e242d1f9e19d27e232cc71e2fa7964fc988a319fce5671ca7f73
The relative path is displayed.