Prometheus settings

Remote read/write

Prometheus supports reading and writing to remote services.

To configure a remote read or write service, you can include the following in gitlab.rb.

prometheus['remote_write'] = [
  {
    url: 'https://some-remote-write-service.example.com',
    basic_auth: {
      password: 'remote write secret password'
    }
  }
]
prometheus['remote_read'] = [
  {
    url: 'https://some-remote-write-service.example.com'
  }
]

For more documentation on the options available, see the remote write and remote read sections of the official documentation.

Rules files

Prometheus allows for recording and alerting rules.

Linux package installations include some default rules files that are stored in /var/opt/gitlab/prometheus/rules/.

To override the default rules, you can change the default list in gitlab.rb..

No rules:

prometheus['rules_files'] = []

Custom list:

prometheus['rules_files'] = ['/path/to/rules/*.rules', '/path/to/single/file.rules']

External labels

To set external labels:

prometheus['external_labels'] = {
    'region' => 'us-west-2',
    'source' => 'omnibus',
}

No external labels are set by default.

node_exporter

The node_exporter provides system level metrics.

Additional metrics collectors are enabled by default. For example, mountstats is used to collect metrics about NFS mounts.

To disable the mountstats collector, adjust gitlab.rb with the following setting and run gitlab-ctl reconfigure:

node_exporter['flags'] = {
  'collector.mountstats' => false,
}

For more information on available collectors, see the upstream documentation.

Alertmanager options

You can set global options for the Alertmanager.

For example, the following gitlab.rb configuration overrides the hostname Alertmanager uses to identify itself to the SMTP server:

alertmanager['global'] = {
  'smtp_hello' => 'example.org'
}

Additional receivers and routes

In this example, we implement a new receiver for VictorOps.

  1. Edit /etc/gitlab/gitlab.rb to add a new receiver and define a route:

    alertmanager['receivers'] = [
  {
    'name' => 'victorOps-receiver',
    'victorops_configs' => [
      {
        'routing_key'         => 'Sample_route',
        'api_key'             => '558e7ebc-XXXX-XXXX-XXXX-XXXXXXXXXXXX',
        'entity_display_name' => '{{ .CommonAnnotations.summary }}',
        'message_type'        => '{{ .CommonLabels.severity }}',
        'state_message'       => 'Alert: {{ .CommonLabels.alertname }}. Summary:{{ .CommonAnnotations.summary }}. RawData: {{ .CommonLabels }}',
        'http_config'         => {
          proxy_url: 'http://internet.proxy.com:3128'
        }
      } #, { Next receiver }
    ]
  }
]

alertmanager['routes'] = [
  {
    'receiver'        => 'victorOps-receiver',
    'group_wait'      => '30s',
    'group_interval'  => '5m',
    'repeat_interval' => '3h',
    'matchers'        => [ 'severity = high' ]
  } #, { Next route }
]

  2. Reconfigure GitLab:

    sudo gitlab-ctl reconfigure

Alertmanager will now route severity = high alerts to victorops-receiver.

Read more about VictorOps options for Alertmanager at the VictorOps documentation.