Configuring GitLab Runner on OpenShift
Offering: GitLab.com, Self-managed
This document explains how to configure GitLab Runner on OpenShift.
Passing properties to GitLab Runner Operator
When creating a Runner
, you can configure it by setting properties in its spec
. For example, you can specify the GitLab URL it will be registered in, or the name of the secret that contains the registration token:
apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
name: dev
spec:
gitlabUrl: https://gitlab.example.com
token: gitlab-runner-secret # Name of the secret containing the Runner token
Read about all the available properties in Operator properties.
Operator properties
This is a list of the supported properties that can be passed to the Operator.
Some properties are only available with more recent versions of the Operator.
Setting | Operator | Description |
---|---|---|
gitlabUrl
| all | The fully qualified domain name for the GitLab instance, for example, https://gitlab.example.com .
|
token
| all | Name of Secret containing the runner-registration-token key used to register the runner.
|
tags
| all | List of comma-separated tags to be applied to the runner. |
concurrent
| all | Limits how many jobs can run concurrently. The maximum number is all defined runners. 0 does not mean unlimited. Default is 10 .
|
interval
| all | Defines the number of seconds between checks for new jobs. Default is 30 .
|
locked
| 1.8 | Defines if the runner should be locked to a project. Default is false .
|
runUntagged
| 1.8 | Defines if jobs without tags should be run. Default is true if no tags were specified. Otherwise, it’s false .
|
protected
| 1.8 | Defines if the runner should run jobs on protected branches only. Default is false .
|
cloneURL
| all | Overwrite the URL for the GitLab instance. Used only if the runner can’t connect to the GitLab URL. |
env
| all | Name of ConfigMap containing key-value pairs that will be injected as environment variables in the Runner pod.
|
runnerImage
| 1.7 | Overwrites the default GitLab Runner image. Default is the Runner image the operator was bundled with. |
helperImage
| all | Overwrites the default GitLab Runner helper image. |
buildImage
| all | The default Docker image to use for builds when none is specified. |
cacheType
| all | Type of cache used for Runner artifacts. One of: gcs , s3 , azure .
|
cachePath
| all | Defines the cache path on the file system. |
cacheShared
| all | Enable sharing of cache between runners. |
s3
| all | Options used to setup S3 cache. Refer to Cache properties. |
gcs
| all | Options used to setup GCS cache. Refer to Cache properties. |
azure
| all | Options used to setup Azure cache. Refer to Cache properties. |
ca
| all | Name of TLS secret containing the custom certificate authority (CA) certificates. |
serviceaccount
| all | Use to override service account used to run the Runner pod. |
config
| all | Use to provide a custom config map with a configuration template. |
Cache properties
S3 cache
Setting | Operator | Description |
---|---|---|
server
| all | The S3 server address. |
credentials
| all | Name of the Secret containing the accesskey and secretkey properties used to access the object storage.
|
bucket
| all | Name of the bucket in which the cache will be stored. |
location
| all | Name of the S3 region which the cache will be stored. |
insecure
| all | Use insecure connections or HTTP .
|
GCS cache
Setting | Operator | Description |
---|---|---|
credentials
| all | Name of the Secret containing the access-id and private-key properties used to access the object storage.
|
bucket
| all | Name of the bucket in which the cache will be stored. |
credentialsFile
| all | Takes GCS credentials file, keys.json .
|
Azure cache
Setting | Operator | Description |
---|---|---|
credentials
| all | Name of the Secret containing the accountName and privateKey properties used to access the object storage.
|
container
| all | Name of the Azure container in which the cache will be stored. |
storageDomain
| all | The domain name of the Azure blob storage. |
Configure a proxy environment
To create a proxy environment:
-
Edit the
custom-env.yaml
file. For example:apiVersion: v1 data: HTTP_PROXY: example.com kind: ConfigMap metadata: name: custom-env
-
Update OpenShift to apply the changes.
oc apply -f custom-env.yaml
-
Update your
gitlab-runner.yml
file.apiVersion: apps.gitlab.com/v1beta2 kind: Runner metadata: name: dev spec: gitlabUrl: https://gitlab.example.com token: gitlab-runner-secret # Name of the secret containing the Runner token env: custom-env
If the proxy can’t reach the Kubernetes API, you might see an error in your CI/CD job:
ERROR: Job failed (system failure): prepare environment: setting up credentials: Post https://172.21.0.1:443/api/v1/namespaces/<KUBERNETES_NAMESPACE>/secrets: net/http: TLS handshake timeout. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
To resolve this error, add the IP address of the Kubernetes API to NO_PROXY
configuration in the custom-env.yaml
file:
apiVersion: v1
data:
NO_PROXY: 172.21.0.1
HTTP_PROXY: example.com
kind: ConfigMap
metadata:
name: custom-env
You can verify the IP address of the Kubernetes API by running:
oc get services --namespace default --field-selector='metadata.name=kubernetes' | grep -v NAME | awk '{print $3}'
Customize config.toml
with a configuration template
config.toml
is currently limited to specifying [runners.kubernetes.volumes]
settings.
Support to extend this to other settings is proposed in issue 49.You can customize the runner’s config.toml
file by using the configuration template.
-
Create a custom config template file. For example, let’s instruct our runner to mount an
EmptyDir
volume. Create thecustom-config.toml
file:[[runners]] [runners.kubernetes] [runners.kubernetes.volumes] [[runners.kubernetes.volumes.empty_dir]] name = "empty-dir" mount_path = "/path/to/empty_dir" medium = "Memory"
-
Create a
ConfigMap
namedcustom-config-toml
from ourcustom-config.toml
file:oc create configmap custom-config-toml --from-file config.toml=custom-config.toml
-
Set the
config
property of theRunner
:apiVersion: apps.gitlab.com/v1beta2 kind: Runner metadata: name: dev spec: gitlabUrl: https://gitlab.example.com token: gitlab-runner-secret config: custom-config-toml
Configure a custom TLS cert
-
To set a custom TLS cert, create a secret with key
tls.crt
. In this example, the file is namedcustom-tls-ca-secret.yaml
:apiVersion: v1 kind: Secret metadata: name: custom-tls-ca type: Opaque stringData: tls.crt: | -----BEGIN CERTIFICATE----- MIIEczCCA1ugAwIBAgIBADANBgkqhkiG9w0BAQQFAD..AkGA1UEBhMCR0Ix ..... 7vQMfXdGsRrXNGRGnX+vWDZ3/zWI0joDtCkNnqEpVn..HoX -----END CERTIFICATE-----
-
Create the secret:
oc apply -f custom-tls-ca-secret.yaml
-
Set the
ca
key in therunner.yaml
to the same name as the name of our secret:apiVersion: apps.gitlab.com/v1beta2 kind: Runner metadata: name: dev spec: gitlabUrl: https://gitlab.example.com token: gitlab-runner-secret ca: custom-tls-ca
Configure the CPU and memory size of runner pods
To set CPU limits and memory limits in a custom config.toml
file, follow the instructions in this topic.
Configure job concurrency per runner based on cluster resources
Set the concurrent
property of the Runner
resource:
apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
name: dev
spec:
gitlabUrl: https://gitlab.example.com
token: gitlab-runner-secret
concurrent: 2
Job concurrency is dictated by the requirements of the project.
- Start by trying to determine the compute and memory resources required to execute a CI job.
- Calculate how many times that job would be able to execute given the resources in the cluster.
If you set too large a concurrency value, the Kubernetes executor will process the jobs as soon as it can. However, the Kubernetes cluster’s scheduler capacity determines when the job is scheduled.
Troubleshooting
Root vs non-root
The GitLab Runner Operator and the GitLab Runner pod run as non-root users. As a result, the build image used in the job would need to run as a non-root user to be able to complete successfully.
This is to ensure that jobs can run successfully with the least permission. However, for this to work,
the build image used for the CI jobs also needs to be built to run as non-root and should not write to
a restricted filesystem. Keep in mind that most container filesystems on an OpenShift cluster will be read-only, except for mounted
volumes, /var/tmp
, /tmp
and other volumes mounted on the root filesystem as tmpfs
.
Overriding the HOME
environment variable
If creating a custom build image or overriding env variables, ensure that the HOME environment variables is not set to /
which would be read-only.
Especially if your jobs would need to write files to the home directory.
You could create a directory under /home
for example /home/ci
and set ENV HOME=/home/ci
in your Dockerfile
.
For the runner pods it’s expected that HOME
would be set to /home/gitlab-runner
.
If this variable is changed, the new location must have the proper permissions.
These guidelines are also documented in the Red Hat Container Platform Docs > Creating Images > Support arbitrary user ids.
Watch out for SCC
By default, when installed in a new OpenShift project, the GitLab Runner Operator will run as non-root.
There are exceptions, when all the service accounts in a project are granted anyuid
access, such as the default
project.
In that case, the user of the image will be root
. This can be easily checked by running the whoami
inside any container shell, e.g. a job.
Read more about SCC in Red Hat Container Platform Docs > Managing security context constraints.
Run As anyuid SCC
Though discouraged, in the event that is it absolutely necessary for a CI job to run as the root
user or to write to the root filesystem, you will need to set the anyuid
SCC on the GitLab Runner
service account, gitlab-runner-sa
, which is used by the GitLab Runner container.
In OpenShift 4.3.8 and earlier:
oc adm policy add-scc-to-user anyuid -z gitlab-runner-sa -n <runner_namespace>
# Check that the anyiud SCC is set:
oc get scc anyuid -o yaml
In OpenShift 4.3.8 and later:
oc create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: scc-anyuid
namespace: <runner_namespace>
rules:
- apiGroups:
- security.openshift.io
resourceNames:
- anyuid
resources:
- securitycontextconstraints
verbs:
- use
EOF
oc create -f - <<EOF
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: sa-to-scc-anyuid
namespace: <runner_namespace>
subjects:
- kind: ServiceAccount
name: gitlab-runner-sa
roleRef:
kind: Role
name: scc-anyuid
apiGroup: rbac.authorization.k8s.io
EOF
Configure SETFCAP
If you use Red Hat OpenShift Container Platform (RHOCP) 4.11 or later, you may get the following error message:
error reading allowed ID mappings:error reading subuid mappings for user
Some jobs (for example, buildah) need the SETFCAP
capability granted to run correctly. To fix this issue:
-
Add the SETFCAP capability to the SCC that GitLab Runner is using (replace the
gitlab-scc
with the SCC assigned to your GitLab Runner pod):oc patch scc gitlab-scc --type merge -p '{"allowedCapabilities":["SETFCAP"]}'
-
Update your
config.toml
and add theSETFCAP
capability under thekubernetes
section:[[runners]] [runners.kubernetes] [runners.kubernetes.pod_security_context] [runners.kubernetes.build_container_security_context] [runners.kubernetes.build_container_security_context.capabilities] add = ["SETFCAP"]
-
Create a configmap using this
config.toml
in the namespace where GitLab Runner is deployed:oc create configmap custom-config-toml --from-file config.toml=config.toml
-
Modify the runner you want to fix, adding the
config:
parameter to point to the recently created configmap (replace my-runner with the correct runner pod name)oc patch runner my-runner --type merge -p '{"spec": {"config": "custom-config-toml"}}'
For more inforamtion, see the Red Hat documentation.
Using FIPS Compliant GitLab Runner
To use a FIPS compliant GitLab Runner Helper, change the helper image as follows:
apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
name: dev
spec:
gitlabUrl: https://gitlab.example.com
token: gitlab-runner-secret
helperImage: gitlab/gitlab-runner-helper:ubi-fips
concurrent: 2
Register GitLab Runner by using a self-signed certificate
When you use a self-signed certificate with your GitLab self-managed installation, you must create a secret that contains the CA certificate used to sign your private certificates.
The name of the secret is then provided as the CA in the Runner spec section:
KIND: Runner
VERSION: apps.gitlab.com/v1beta2
FIELD: ca <string>
DESCRIPTION:
Name of tls secret containing the custom certificate authority (CA)
certificates
The secret can be created using the following command:
oc create secret generic mySecret --from-file=tls.crt=myCert.pem -o yaml
Register GitLab Runner with an external URL that points to an IP address
If the runner cannot match the self-signed certificate with the hostname, you might get an error message. This can happen when the GitLab self-managed instance is configured to be accessed from an IP address instead of a hostname (where ###.##.##.## is the IP address of the GitLab server):
[31;1mERROR: Registering runner... failed [0;m [31;1mrunner[0;m=A5abcdEF [31;1mstatus[0;m=couldn't execute POST against https://###.##.##.##/api/v4/runners:
Post https://###.##.##.##/api/v4/runners: x509: cannot validate certificate for ###.##.##.## because it doesn't contain any IP SANs
[31;1mPANIC: Failed to register the runner. You may be having network problems.[0;m
To fix this issue:
-
On the GitLab self-managed server, modify the
openssl
to add the IP address to thesubjectAltName
parameter:# vim /etc/pki/tls/openssl.cnf [ v3_ca ] subjectAltName=IP:169.57.64.36 <---- Add this line. 169.57.64.36 is your GitLab server IP.
-
Then re-generate a self-signed CA with the commands below:
# cd /etc/gitlab/ssl # openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -keyout /etc/gitlab/ssl/169.57.64.36.key -out /etc/gitlab/ssl/169.57.64.36.crt # openssl dhparam -out /etc/gitlab/ssl/dhparam.pem 4096 # gitlab-ctl restart
-
Use this new certificate to generate a new secret.