Workhorse configuration
For historical reasons, Workhorse uses:
- Command line flags.
- A configuration file.
- Environment variables.
Add any new Workhorse configuration options into the configuration file.
CLI options
gitlab-workhorse [OPTIONS]
Options:
-apiCiLongPollingDuration duration
Long polling duration for job requesting for runners (default 50ns)
-apiLimit uint
Number of API requests allowed at single time
-apiQueueDuration duration
Maximum queueing duration of requests (default 30s)
-apiQueueLimit uint
Number of API requests allowed to be queued
-authBackend string
Authentication/authorization backend (default "http://localhost:8080")
-authSocket string
Optional: Unix domain socket to dial authBackend at
-cableBackend string
ActionCable backend
-cableSocket string
Optional: Unix domain socket to dial cableBackend at
-config string
TOML file to load config from
-developmentMode
Allow the assets to be served from Rails app
-documentRoot string
Path to static files content (default "public")
-listenAddr string
Listen address for HTTP server (default "localhost:8181")
-listenNetwork string
Listen 'network' (tcp, tcp4, tcp6, unix) (default "tcp")
-listenUmask int
Umask for Unix socket
-logFile string
Log file location
-logFormat string
Log format to use defaults to text (text, json, structured, none) (default "text")
-pprofListenAddr string
pprof listening address, e.g. 'localhost:6060'
-prometheusListenAddr string
Prometheus listening address, e.g. 'localhost:9229'
-propagateCorrelationID X-Request-ID
Reuse existing Correlation-ID from the incoming request header X-Request-ID if present
-proxyHeadersTimeout duration
How long to wait for response headers when proxying the request (default 5m0s)
-secretPath string
File with secret key to authenticate with authBackend (default "./.gitlab_workhorse_secret")
-version
Print version and exit
The ‘auth backend’ refers to the GitLab Rails application. The name is
a holdover from when GitLab Workhorse only handled git push
and git pull
over
HTTP.
GitLab Workhorse can listen on either a TCP or a Unix domain socket. It
can also open a second listening TCP listening socket with the Go
net/http/pprof
profiler server.
GitLab Workhorse can listen on Redis build and runner registration events if you
pass a valid TOML configuration file through the -config
flag.
A regular setup it only requires the following (replacing the string
with the actual socket)
Redis
GitLab Workhorse integrates with Redis to do long polling for CI build requests. To configure it:
- Configure Redis settings in the TOML configuration file.
- Control polling behavior for CI build requests with the
-apiCiLongPollingDuration
command-line flag.
You can enable Redis in the configuration file while leaving CI polling disabled. This configuration results in an idle Redis Pub/Sub connection. The opposite is not possible: CI long polling requires a correct Redis configuration.
For example, the [redis]
section in the configuration file could contain:
[redis]
URL = "unix:///var/run/gitlab/redis.sock"
Password = "my_awesome_password"
Sentinel = [ "tcp://sentinel1:23456", "tcp://sentinel2:23456" ]
SentinelMaster = "mymaster"
-
URL
- A string in the formatunix://path/to/redis.sock
ortcp://host:port
. -
Password
- Required only if your Redis instance is password-protected. -
Sentinel
- Required if you use Sentinel.
If both Sentinel
and URL
are given, only Sentinel
is used.
Optional fields:
[redis]
DB = 0
MaxIdle = 1
MaxActive = 1
-
DB
- The database to connect to. Defaults to0
. -
MaxIdle
- How many idle connections can be in the Redis pool at once. Defaults to1
. -
MaxActive
- How many connections the pool can keep. Defaults to1
.
Relative URL support
If you mount GitLab at a relative URL, like example.com/gitlab
), use this
relative URL in the authBackend
setting:
gitlab-workhorse -authBackend http://localhost:8080/gitlab
TLS support
A listener with TLS can be configured to be used for incoming requests. Paths to the files containing a certificate and matching private key for the server must be provided:
[[listeners]]
network = "tcp"
addr = "localhost:3443"
[listeners.tls]
certificate = "/path/to/certificate"
key = "/path/to/private/key"
min_version = "tls1.2"
max_version = "tls1.3"
The certificate
file should contain the concatenation
of the server’s certificate, any intermediates, and the CA’s certificate.
Metrics endpoints can be configured similarly:
[metrics_listener]
network = "tcp"
addr = "localhost:9229"
[metrics_listener.tls]
certificate = "/path/to/certificate"
key = "/path/to/private/key"
min_version = "tls1.2"
max_version = "tls1.3"
Interaction of authBackend
and authSocket
The interaction between authBackend
and authSocket
can be confusing.
If authSocket
is set, it overrides the host portion of authBackend
, but not
the relative path.
In table form:
authBackend | authSocket | Workhorse connects to | Rails relative URL |
---|---|---|---|
unset | unset | localhost:8080 | / |
http://localhost:3000 | unset | localhost:3000 | / |
http://localhost:3000/gitlab | unset | localhost:3000 | /gitlab |
unset | /path/to/socket | /path/to/socket | / |
http://localhost:3000 | /path/to/socket | /path/to/socket | / |
http://localhost:3000/gitlab | /path/to/socket | /path/to/socket | /gitlab |
The same applies to cableBackend
and cableSocket
.
Error tracking
GitLab-Workhorse supports remote error tracking with Sentry.
To enable this feature, set the GITLAB_WORKHORSE_SENTRY_DSN
environment variable.
You can also set the GITLAB_WORKHORSE_SENTRY_ENVIRONMENT
environment variable to
use the Sentry environment feature to separate staging, production and
development.
Omnibus GitLab (/etc/gitlab/gitlab.rb
):
gitlab_workhorse['env'] = {
'GITLAB_WORKHORSE_SENTRY_DSN' => 'https://foobar'
'GITLAB_WORKHORSE_SENTRY_ENVIRONMENT' => 'production'
}
Source installations (/etc/default/gitlab
):
export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar'
export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production'
Distributed tracing
Workhorse supports distributed tracing through LabKit using OpenTracing APIs.
By default, no tracing implementation is linked into the binary. You can link in
different OpenTracing providers with build tags
or build constraints by setting the BUILD_TAGS
make variable.
For more details of the supported providers, refer to LabKit. For an example of
Jaeger tracing support, include the tags: BUILD_TAGS="tracer_static tracer_static_jaeger"
like this:
make BUILD_TAGS="tracer_static tracer_static_jaeger"
After you compile Workhorse with an OpenTracing provider, configure the tracing
configuration with the GITLAB_TRACING
environment variable, like this:
GITLAB_TRACING=opentracing://jaeger ./gitlab-workhorse
Propagate correlation IDs
When a user makes an HTTP request, such as creating a new project, the
initial request is routed through Workhorse to another service, which
may in turn, make other requests. To help trace the request as it flows
across services, Workhorse generates a random value called a
correlation ID.
Workhorse sends this correlation ID via the X-Request-Id
HTTP header.
Some GitLab services, such as GitLab Shell, generate their own
correlation IDs. In addition, other services, such as Gitaly, make
internal API calls that pass along a correlation ID from the original
request. In either case, the correlation ID is also passed via the
X-Request-Id
HTTP header.
By default, Workhorse ignores this header and always generates a new correlation ID. This makes debugging harder and prevents distributed tracing from working properly, since the new correlation ID is completely unrelated to the original one.
Workhorse can be configured to propagate an incoming correlation ID via
the -propagateCorrelationID
command-line flag. It is highly
recommended that this option be used with an IP allow list to ensure
arbitrary values cannot be generated by untrusted clients.
An IP allow list is specified via the trusted_cidrs_for_propagation
option in the Workhorse configuration file. Specify a list of CIDR blocks
that can be trusted. For example:
trusted_cidrs_for_propagation = ["10.0.0.0/8", "127.0.0.1/32"]
-propagateCorrelationID
flag must be used for the trusted_cidrs_for_propagation
option to work.Trusted proxies
If Workhorse is behind a reverse proxy such as NGINX, the
trusted_cidrs_for_x_forwarded_for
option is needed to specify which
CIDR blocks can be used to trust to provide the originating IP address
via the X-Forwarded-For
HTTP header. For example:
trusted_cidrs_for_x_forwarded_for = ["10.0.0.0/8", "127.0.0.1/32"]
Continuous profiling
Workhorse supports continuous profiling through LabKit using Stackdriver Profiler. By default, the Stackdriver Profiler implementation is linked in the binary using build tags, though it’s not required and can be skipped. For example:
make BUILD_TAGS=""
After you compile Workhorse with continuous profiling, set the profiler configuration
with the GITLAB_CONTINUOUS_PROFILING
environment variable. For example:
GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1&project_id=test-123 ./gitlab-workhorse"