Configuring GitLab Runner on OpenShift

This document explains how to configure GitLab Runner on OpenShift.

Passing properties to GitLab Runner Operator

When creating a Runner, you can configure it by setting properties in its spec. For example, you can specify the GitLab URL it will be registered in, or the name of the secret that contains the registration token:

apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
 name: dev
spec:
 gitlabUrl: https://gitlab.example.com
 token: gitlab-runner-secret # Name of the secret containing the Runner token

Read about all the available properties in Operator properties.

Operator properties

This is a list of the supported properties that can be passed to the Operator.

Some properties are only available with more recent versions of the Operator.

SettingOperatorDescription
gitlabUrlallThe fully qualified domain name for the GitLab instance, for example, https://gitlab.example.com.
tokenallName of Secret containing the runner-registration-token key used to register the runner.
tagsallList of comma-separated tags to be applied to the runner.
concurrentallLimits how many jobs can run concurrently. The maximum number is all defined runners. 0 does not mean unlimited. Default is 10.
intervalallDefines the number of seconds between checks for new jobs. Default is 30.
locked1.8Defines if the runner should be locked to a project. Default is false.
runUntagged1.8Defines if jobs without tags should be run. Default is true if no tags were specified. Otherwise, it’s false.
protected1.8Defines if the runner should run jobs on protected branches only. Default is false.
cloneURLallOverwrite the URL for the GitLab instance. Used only if the runner can’t connect to the GitLab URL.
envallName of ConfigMap containing key-value pairs that will be injected as environment variables in the Runner pod.
runnerImage1.7Overwrites the default GitLab Runner image. Default is the Runner image the operator was bundled with.
helperImageallOverwrites the default GitLab Runner helper image.
buildImageallThe default Docker image to use for builds when none is specified.
cacheTypeallType of cache used for Runner artifacts. One of: gcs, s3, azure.
cachePathallDefines the cache path on the file system.
cacheSharedallEnable sharing of cache between runners.
s3allOptions used to setup S3 cache. Refer to Cache properties.
gcsallOptions used to setup GCS cache. Refer to Cache properties.
azureallOptions used to setup Azure cache. Refer to Cache properties.
caallName of TLS secret containing the custom certificate authority (CA) certificates.
serviceaccountallUse to override service account used to run the Runner pod.
configallUse to provide a custom config map with a configuration template.

Cache properties

S3 cache

SettingOperatorDescription
serverallThe S3 server address.
credentialsallName of the Secret containing the accesskey and secretkey properties used to access the object storage.
bucketallName of the bucket in which the cache will be stored.
locationallName of the S3 region which the cache will be stored.
insecureallUse insecure connections or HTTP.

GCS cache

SettingOperatorDescription
credentialsallName of the Secret containing the access-id and private-key properties used to access the object storage.
bucketallName of the bucket in which the cache will be stored.
credentialsFileallTakes GCS credentials file, keys.json.

Azure cache

SettingOperatorDescription
credentialsallName of the Secret containing the accountName and privateKey properties used to access the object storage.
containerallName of the Azure container in which the cache will be stored.
storageDomainallThe domain name of the Azure blob storage.

Configure a proxy environment

To create a proxy environment:

  1. Edit the custom-env.yaml file. For example:

    apiVersion: v1
    data:
      HTTP_PROXY: example.com
    kind: ConfigMap
    metadata:
      name: custom-env
    
  2. Update OpenShift to apply the changes.

    oc apply -f custom-env.yaml
    
  3. Update your gitlab-runner.yml file.

    apiVersion: apps.gitlab.com/v1beta2
    kind: Runner
    metadata:
      name: dev
    spec:
      gitlabUrl: https://gitlab.example.com
      token: gitlab-runner-secret # Name of the secret containing the Runner token
      env: custom-env
    

If the proxy can’t reach the Kubernetes API, you might see an error in your CI/CD job:

ERROR: Job failed (system failure): prepare environment: setting up credentials: Post https://172.21.0.1:443/api/v1/namespaces/<KUBERNETES_NAMESPACE>/secrets: net/http: TLS handshake timeout. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

To resolve this error, add the IP address of the Kubernetes API to NO_PROXY configuration in the custom-env.yaml file:

   apiVersion: v1
   data:
     NO_PROXY: 172.21.0.1
     HTTP_PROXY: example.com
   kind: ConfigMap
   metadata:
     name: custom-env

You can verify the IP address of the Kubernetes API by running:

oc get services --namespace default --field-selector='metadata.name=kubernetes' | grep -v NAME | awk '{print $3}'

Customize config.toml with a configuration template

note
The use of a configuration template to customize config.toml is currently limited to specifying [runners.kubernetes.volumes] settings. Support to extend this to other settings is proposed in issue 49.

You can customize the runner’s config.toml file by using the configuration template.

  1. Create a custom config template file. For example, let’s instruct our runner to mount an EmptyDir volume. Create the custom-config.toml file:

    [[runners]]
      [runners.kubernetes]
        [runners.kubernetes.volumes]
          [[runners.kubernetes.volumes.empty_dir]]
            name = "empty-dir"
            mount_path = "/path/to/empty_dir"
            medium = "Memory"
    
  2. Create a ConfigMap named custom-config-toml from our custom-config.toml file:

     oc create configmap custom-config-toml --from-file config.toml=custom-config.toml
    
  3. Set the config property of the Runner:

     apiVersion: apps.gitlab.com/v1beta2
     kind: Runner
     metadata:
       name: dev
     spec:
       gitlabUrl: https://gitlab.example.com
       token: gitlab-runner-secret
       config: custom-config-toml
    

Configure a custom TLS cert

  1. To set a custom TLS cert, create a secret with key tls.crt. In this example, the file is named custom-tls-ca-secret.yaml:

     apiVersion: v1
     kind: Secret
     metadata:
         name: custom-tls-ca
     type: Opaque
     stringData:
         tls.crt: |
             -----BEGIN CERTIFICATE-----
             MIIEczCCA1ugAwIBAgIBADANBgkqhkiG9w0BAQQFAD..AkGA1UEBhMCR0Ix
             .....
             7vQMfXdGsRrXNGRGnX+vWDZ3/zWI0joDtCkNnqEpVn..HoX
             -----END CERTIFICATE-----
    
  2. Create the secret:

    oc apply -f custom-tls-ca-secret.yaml
    
  3. Set the ca key in the runner.yaml to the same name as the name of our secret:

     apiVersion: apps.gitlab.com/v1beta2
     kind: Runner
     metadata:
       name: dev
     spec:
       gitlabUrl: https://gitlab.example.com
       token: gitlab-runner-secret
       ca: custom-tls-ca
    

Configure the CPU and memory size of runner pods

To set CPU limits and memory limits in a custom config.toml file, follow the instructions in this topic.

Configure job concurrency per runner based on cluster resources

Set the concurrent property of the Runner resource:

   apiVersion: apps.gitlab.com/v1beta2
   kind: Runner
   metadata:
     name: dev
   spec:
     gitlabUrl: https://gitlab.example.com
     token: gitlab-runner-secret
     concurrent: 2

Job concurrency is dictated by the requirements of the project.

  1. Start by trying to determine the compute and memory resources required to execute a CI job.
  2. Calculate how many times that job would be able to execute given the resources in the cluster.

If you set too large a concurrency value, the Kubernetes executor will process the jobs as soon as it can. However, the Kubernetes cluster’s scheduler capacity determines when the job is scheduled.

Troubleshooting

Root vs non-root

The GitLab Runner Operator and the GitLab Runner pod run as non-root users. As a result, the build image used in the job would need to run as a non-root user to be able to complete successfully. This is to ensure that jobs can run successfully with the least permission. However, for this to work, the build image used for the CI jobs also needs to be built to run as non-root and should not write to a restricted filesystem. Keep in mind that most container filesystems on an OpenShift cluster will be read-only, except for mounted volumes, /var/tmp, /tmp and other volumes mounted on the root filesystem as tmpfs.

Overriding the HOME environment variable

If creating a custom build image or overriding env variables, ensure that the HOME environment variables is not set to / which would be read-only. Especially if your jobs would need to write files to the home directory. You could create a directory under /home for example /home/ci and set ENV HOME=/home/ci in your Dockerfile.

For the runner pods it’s expected that HOME would be set to /home/gitlab-runner. If this variable is changed, the new location must have the proper permissions. These guidelines are also documented in the Red Hat Container Platform Docs > Creating Images > Support arbitrary user ids.

Watch out for SCC

By default, when installed in a new OpenShift project, the GitLab Runner Operator will run as non-root. There are exceptions, when all the service accounts in a project are granted anyuid access, such as the default project. In that case, the user of the image will be root. This can be easily checked by running the whoami inside any container shell, e.g. a job. Read more about SCC in Red Hat Container Platform Docs > Managing security context constraints.

Run As anyuid SCC

Though discouraged, in the event that is it absolutely necessary for a CI job to run as the root user or to write to the root filesystem, you will need to set the anyuid SCC on the GitLab Runner service account, gitlab-runner-sa, which is used by the GitLab Runner container.

In OpenShift 4.3.8 and earlier:

oc adm policy add-scc-to-user anyuid -z gitlab-runner-sa -n <runner_namespace>

# Check that the anyiud SCC is set:
oc get scc anyuid -o yaml

In OpenShift 4.3.8 and later:

oc create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: scc-anyuid
  namespace: <runner_namespace>
rules:
- apiGroups:
  - security.openshift.io
  resourceNames:
  - anyuid
  resources:
  - securitycontextconstraints
  verbs:
  - use
EOF

oc create -f - <<EOF
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sa-to-scc-anyuid
  namespace: <runner_namespace>
subjects:
  - kind: ServiceAccount
    name: gitlab-runner-sa
roleRef:
  kind: Role
  name: scc-anyuid
  apiGroup: rbac.authorization.k8s.io
EOF

Configure SETFCAP

If you use Red Hat OpenShift Container Platform (RHOCP) 4.11 or later, you may get the following error message:

error reading allowed ID mappings:error reading subuid mappings for user

Some jobs (for example, buildah) need the SETFCAP capability granted to run correctly. To fix this issue:

  1. Add the SETFCAP capability to the SCC that GitLab Runner is using (replace the gitlab-scc with the SCC assigned to your GitLab Runner pod):

     oc patch scc gitlab-scc --type merge -p '{"allowedCapabilities":["SETFCAP"]}'  
    
  2. Update your config.toml and add the SETFCAP capability under the kubernetes section:

     [[runners]]
       [runners.kubernetes]
       [runners.kubernetes.pod_security_context]
         [runners.kubernetes.build_container_security_context]
         [runners.kubernetes.build_container_security_context.capabilities]
           add = ["SETFCAP"]
    
  3. Create a configmap using this config.toml in the namespace where GitLab Runner is deployed:

     oc create configmap custom-config-toml --from-file config.toml=config.toml 
    
  4. Modify the runner you want to fix, adding the config: parameter to point to the recently created configmap (replace my-runner with the correct runner pod name)

     oc patch runner my-runner --type merge -p '{"spec": {"config": "custom-config-toml"}}'
    

For more inforamtion, see the Red Hat documentation.

Using FIPS Compliant GitLab Runner

note
Currently, for Operator, you can change only the helper image. An issue exists to change the GitLab Runner image as well.

To use a FIPS compliant GitLab Runner Helper, change the helper image as follows:

apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
 name: dev
spec:
 gitlabUrl: https://gitlab.example.com
 token: gitlab-runner-secret
 helperImage: gitlab/gitlab-runner-helper:ubi-fips
 concurrent: 2

Register GitLab Runner by using a self-signed certificate

When you use a self-signed certificate with your GitLab self-managed installation, you must create a secret that contains the CA certificate used to sign your private certificates.

The name of the secret is then provided as the CA in the Runner spec section:

KIND:     Runner
VERSION:  apps.gitlab.com/v1beta2

FIELD:    ca <string>

DESCRIPTION:
     Name of tls secret containing the custom certificate authority (CA)
     certificates

The secret can be created using the following command:

oc create secret generic mySecret --from-file=tls.crt=myCert.pem -o yaml

Register GitLab Runner with an external URL that points to an IP address

If the runner cannot match the self-signed certificate with the hostname, you might get an error message. This can happen when the GitLab self-managed instance is configured to be accessed from an IP address instead of a hostname (where ###.##.##.## is the IP address of the GitLab server):

[31;1mERROR: Registering runner... failed               [0;m  [31;1mrunner[0;m=A5abcdEF [31;1mstatus[0;m=couldn't execute POST against https://###.##.##.##/api/v4/runners:
Post https://###.##.##.##/api/v4/runners: x509: cannot validate certificate for ###.##.##.## because it doesn't contain any IP SANs
[31;1mPANIC: Failed to register the runner. You may be having network problems.[0;m

To fix this issue:

  1. On the GitLab self-managed server, modify the openssl to add the IP address to the subjectAltName parameter:

    # vim /etc/pki/tls/openssl.cnf
    
    [ v3_ca ]
    subjectAltName=IP:169.57.64.36 <---- Add this line. 169.57.64.36 is your GitLab server IP.
    
  2. Then re-generate a self-signed CA with the commands below:

    # cd /etc/gitlab/ssl
    # openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -keyout /etc/gitlab/ssl/169.57.64.36.key -out /etc/gitlab/ssl/169.57.64.36.crt
    # openssl dhparam -out /etc/gitlab/ssl/dhparam.pem 4096
    # gitlab-ctl restart
    
  3. Use this new certificate to generate a new secret.