- 1. Install dependencies and Go runtime
- 2. Install Docker Engine
- 3. Download GitLab Runner sources
- 4. Install GitLab Runner dependencies
- 5. Run GitLab Runner
- 6. Run test suite locally
- 7. Run tests with helper image version of choice
- 8. Install optional tools
- 9. Contribute
- Managing build dependencies
- Tests
- Developing for Windows on a non-windows environment
- Other resources
- Troubleshooting
Development environment
1. Install dependencies and Go runtime
For Debian/Ubuntu
sudo apt-get install -y mercurial git-core wget make build-essential
wget https://storage.googleapis.com/golang/go1.17.7.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go*-*.tar.gz
For CentOS
sudo yum install mercurial wget make
sudo yum groupinstall 'Development Tools'
wget https://storage.googleapis.com/golang/go1.17.7.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go*-*.tar.gz
For macOS
Using binary package:
wget https://storage.googleapis.com/golang/go1.17.7.darwin-amd64.tar.gz
sudo tar -C /usr/local -xzf go*-*.tar.gz
Using installation package:
wget https://storage.googleapis.com/golang/go1.17.7.darwin-amd64.pkg
open go*-*.pkg
For FreeBSD
pkg install go-1.17.7 gmake git mercurial
2. Install Docker Engine
The Docker Engine is required to create pre-built image that is embedded into GitLab Runner and loaded when using Docker executor.
To install Docker, follow the Docker installation instructions for your OS.
Make sure you have a binfmt_misc
on the machine that is running your Docker Engine.
This is required for building ARM images that are embedded into the GitLab Runner binary.
-
For Debian/Ubuntu it’s sufficient to execute:
sudo apt-get install binfmt-support qemu-user-static
-
For Docker for MacOS/Windows
binfmt_misc
is enabled by default. -
For CoreOS (but also works on Debian and Ubuntu) you need to execute the following script on system start:
#!/bin/sh set -xe /sbin/modprobe binfmt_misc mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc # Support for ARM binaries through Qemu: { echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:' > /proc/sys/fs/binfmt_misc/register; } 2>/dev/null { echo ':armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-armeb-static:' > /proc/sys/fs/binfmt_misc/register; } 2>/dev/null { echo ':aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:CF' > /proc/sys/fs/binfmt_misc/register; } 2>/dev/null
3. Download GitLab Runner sources
go get gitlab.com/gitlab-org/gitlab-runner
4. Install GitLab Runner dependencies
After you clone GitLab Runner, cd
into the gitlab-runner
directory and download and restore all build dependencies:
git clone <gitlab-runner-uri>
cd gitlab-runner
make deps
For FreeBSD use gmake deps
5. Run GitLab Runner
Normally you would use gitlab-runner
. In order to compile and run the Go sources, use the Go toolchain:
make runner-and-helper-bin-host
./out/binaries/gitlab-runner run
You can run GitLab Runner in debug-mode:
make runner-and-helper-bin-host
./out/binaries/gitlab-runner --debug run
make runner-and-helper-bin-host
is a superset of make runner-bin-host
which in addition
takes care of building the Runner Helper Docker archive dependencies.
Building the Docker images
If you want to build the Docker images, run make runner-and-helper-docker-host
, which will:
- Build
gitlab-runner-helper
and create a helper Docker image from it. - Compile GitLab Runner for
linux/amd64
. - Build a DEB package for Runner. The official GitLab Runner images are based on Alpine and Ubuntu, and the Ubuntu image build uses the DEB package.
- Build the Alpine and Ubuntu versions of the
gitlab/gitlab-runner
image.
6. Run test suite locally
GitLab Runner test suite consists of “core” tests and tests for executors. Tests for executors require certain binaries to be installed on your local machine. Some of these binaries cannot be installed on all operating systems. If a binary is not installed tests requiring this binary will be skipped.
These are the binaries that you can install:
- VirtualBox and Vagrant; the Vagrant Parallels plugin is also required
- kubectl with minikube
- Parallels Pro or Business edition
- PowerShell
After installing the binaries run:
make development_setup
To execute the tests run:
make test
7. Run tests with helper image version of choice
If you are developing functionality inside a helper, you’ll most likely want to run tests with the version of the Docker image that contains the newest changes.
If you run tests without passing -ldflags
, the default version in version.go
is development
.
This means that the runner defaults to pulling a helper image
with the latest
tag.
Make targets
make
targets inject -ldflags
automatically. You can run all tests by using:
make simple-test
make
targets also inject -ldflags
for parallel_test_execute
, which is most commonly used by the CI/CD jobs.
Custom go test
arguments
In case you want a more customized go test
command, you can use print_ldflags
as make
target:
go test -ldflags "$(make print_ldflags)" -run TestDockerCommandBuildCancel -v ./executors/docker/...
In GoLand
Currently, GoLand doesn’t support dynamic Go tool arguments, so you’ll need to run make print_ldflags
first
and then paste it in the configuration.
-s -w
).Helper image
Build the newest version of the helper image with:
make helper-dockerarchive-host
Then you’ll have the image ready for use:
REPOSITORY TAG IMAGE ID CREATED SIZE
gitlab/gitlab-runner-helper x86_64-a6bc0800 f10d9b5bbb41 32 seconds ago 57.2MB
Helper image with Kubernetes
If you are running a local Kubernetes cluster make sure to reuse the cluster’s Docker daemon to build images. For example, with minikube:
eval $(minikube docker-env)
8. Install optional tools
- Install
golangci-lint
, used for themake lint
target. - Install
markdown-lint
andvale
, used for themake lint-docs
target.
Installation instructions will pop up when running a Makefile target if a tool is missing.
9. Contribute
You can start hacking gitlab-runner
code.
If you need an IDE to edit and debug code, there are a few free suggestions you can use:
- JetBrains GoLand IDE.
- Visual Studio Code using the
workspace recommended extensions,
located in
.vscode/extensions.json
.
Managing build dependencies
GitLab Runner uses Go Modules to manage its dependencies.
Don’t add dependency from upstream default branch when version tags are available.
Tests
The Runner codebase makes a distinction between unit and integration tests in the following way:
-
Unit test files have a suffix of
_test.go
and contain the following build directive in the header:// +build !integration
-
Integration test files have a suffix of
_integration_test.go
and contain the following build directive in the header:// +build integration
They can be run by adding
-tags=integration
to thego test
command.
To test the state of the build directives in test files, make check_test_directives
can be used.
Developing for Windows on a non-windows environment
We provide a Vagrantfile to help you run a Windows Server 2019 or Windows 10 instance, since we are using multiple machines inside of Vagrant.
The following are required:
- Vagrant installed.
- Virtualbox installed.
- Around 30GB of free hard disk space on your computer.
Which virtual machine to use depends on your use case:
- The Windows Server machine has Docker pre-installed and should always be used when you are developing on GitLab Runner for Windows.
- The Windows 10 machine is there for you to have a windows environment with a GUI which sometimes can help you debugging some Windows features. Note that you cannot have Docker running inside of Windows 10 because nested virtualization is not supported.
Running vagrant up windows_10
will start the Windows 10 machine for
you. To:
- SSH inside of the Windows 10 machine, run
vagrant ssh windows_10
. - Access the GUI for the Windows 10, you can connect via
RDP by running
vagrant rdp windows_10
, which will connect to the machine using a locally installed RDP program.
For both machines, the GitLab Runner source code is synced
bi-directionally so that you can edit from your machine with your
favorite editor. The source code can be found under the $GOROOT
environment variable. We have a RUNNER_SRC
environment variable which
you can use to find out the full path so when using PowerShell,
you can use cd $Env:RUNNER_SRC
.
Other resources
- Reviewing GitLab Runner merge requests
- Add support for new Windows Version
- Runner Group - Team Resources
Troubleshooting
docker.go missing Asset symbol
This error happens due to missing executors/docker/bindata.go
file that is generated from Docker prebuilts.
Which is especially tricky on Windows.
Try to execute: make deps docker
, if it doesn’t help you can do that in steps:
- Execute
go get -u github.com/jteeuwen/go-bindata/...
- Download https://gitlab-runner-downloads.s3.amazonaws.com/main/docker/prebuilt-x86_64.tar.xz and save to
out/docker/prebuilt-x86_64.tar.xz
- Download https://gitlab-runner-downloads.s3.amazonaws.com/main/docker/prebuilt-arm.tar.xz and save to
out/docker/prebuilt-arm.tar.xz
- Execute
make docker
or check the Makefile how this command looks like