GitLab Prometheus metrics

To enable the GitLab Prometheus metrics:

  1. Log in to GitLab as a user with administrator access.
  2. On the left sidebar, select Search or go to.
  3. Select Admin Area.
  4. On the left sidebar, select Settings > Metrics and profiling.
  5. Find the Metrics - Prometheus section, and select Enable GitLab Prometheus metrics endpoint.
  6. Restart GitLab for the changes to take effect.

For self-compiled installations, you must configure it yourself.

Collecting the metrics

GitLab monitors its own internal service metrics, and makes them available at the /-/metrics endpoint. Unlike other Prometheus exporters, to access the metrics, the client IP address must be explicitly allowed.

These metrics are enabled and collected for Linux package and Helm chart installations. For self-compiled installations, these metrics must be enabled manually and collected by a Prometheus server.

For enabling and viewing metrics from Sidekiq nodes, see Sidekiq metrics.

Metrics available

caller_id removed from redis_hit_miss_operations_total and redis_cache_generation_duration_seconds in GitLab 15.11.

The following metrics are available:

MetricTypeSinceDescriptionLabels
gitlab_cache_misses_totalCounter10.2Cache read miss controller, action, store
gitlab_cache_operation_duration_secondsHistogram10.2Cache access time operation, store
gitlab_cache_operations_totalCounter12.2Cache operations by controller or action controller, action, operation, store
gitlab_cache_read_multikey_countHistogram15.7Count of keys in multi-key cache read operations controller, action, store
gitlab_ci_pipeline_builder_scoped_variables_durationHistogram14.5Time in seconds it takes to create the scoped variables for a CI/CD job 
gitlab_ci_pipeline_creation_duration_secondsHistogram13.0Time in seconds it takes to create a CI/CD pipelinegitlab
gitlab_ci_pipeline_size_buildsHistogram13.1Total number of builds within a pipeline grouped by a pipeline sourcesource
gitlab_ci_runner_authentication_success_totalCounter15.2Total number of times that runner authentication has succeededtype
gitlab_ci_runner_authentication_failure_totalCounter15.2Total number of times that runner authentication has failed 
gitlab_ghost_user_migration_lag_secondsGauge15.6The waiting time in seconds of the oldest scheduled record for ghost user migration 
gitlab_ghost_user_migration_scheduled_records_totalGauge15.6The total number of scheduled ghost user migrations 
job_waiter_started_totalCounter12.9Number of batches of jobs started where a web request is waiting for the jobs to completeworker
job_waiter_timeouts_totalCounter12.9Number of batches of jobs that timed out where a web request is waiting for the jobs to completeworker
gitlab_ci_active_jobsHistogram14.2Count of active jobs when pipeline is created 
gitlab_database_transaction_secondsHistogram12.1Time spent in database transactions, in seconds 
gitlab_method_call_duration_secondsHistogram10.2Method calls real duration controller, action, module, method
gitlab_omniauth_login_totalCounter16.1Total number of OmniAuth logins attempts omniauth_provider, status
gitlab_page_out_of_boundsCounter12.8Counter for the PageLimiter pagination limit being hit controller, action, bot
gitlab_rails_boot_time_secondsGauge14.8Time elapsed for Rails primary process to finish startup 
gitlab_rails_queue_duration_secondsHistogram9.4Measures latency between GitLab Workhorse forwarding a request to Rails 
gitlab_sql_duration_secondsHistogram10.2SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT  
gitlab_sql_<role>_duration_secondsHistogram13.10SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT, grouped by database roles (primary/replica) 
gitlab_ruby_threads_max_expected_threadsGauge13.3Maximum number of threads expected to be running and performing application work 
gitlab_ruby_threads_running_threadsGauge13.3Number of running Ruby threads by name 
gitlab_transaction_cache_<key>_count_totalCounter10.2Counter for total Rails cache calls (per key) 
gitlab_transaction_cache_<key>_duration_totalCounter10.2Counter for total time (seconds) spent in Rails cache calls (per key) 
gitlab_transaction_cache_count_totalCounter10.2Counter for total Rails cache calls (aggregate) 
gitlab_transaction_cache_duration_totalCounter10.2Counter for total time (seconds) spent in Rails cache calls (aggregate) 
gitlab_transaction_cache_read_hit_count_totalCounter10.2Counter for cache hits for Rails cache calls controller, action, store
gitlab_transaction_cache_read_miss_count_totalCounter10.2Counter for cache misses for Rails cache calls controller, action, store
gitlab_transaction_duration_secondsHistogram10.2Duration for successful requests (gitlab_transaction_* metrics) controller, action
gitlab_transaction_event_build_found_totalCounter9.4Counter for build found for API /jobs/request 
gitlab_transaction_event_build_invalid_totalCounter9.4Counter for build invalid due to concurrency conflict for API /jobs/request 
gitlab_transaction_event_build_not_found_cached_totalCounter9.4Counter for cached response of build not found for API /jobs/request 
gitlab_transaction_event_build_not_found_totalCounter9.4Counter for build not found for API /jobs/request 
gitlab_transaction_event_change_default_branch_totalCounter9.4Counter when default branch is changed for any repository 
gitlab_transaction_event_create_repository_totalCounter9.4Counter when any repository is created 
gitlab_transaction_event_etag_caching_cache_hit_totalCounter9.4Counter for ETag cache hit.endpoint
gitlab_transaction_event_etag_caching_header_missing_totalCounter9.4Counter for ETag cache miss - header missingendpoint
gitlab_transaction_event_etag_caching_key_not_found_totalCounter9.4Counter for ETag cache miss - key not foundendpoint
gitlab_transaction_event_etag_caching_middleware_used_totalCounter9.4Counter for ETag middleware accessedendpoint
gitlab_transaction_event_etag_caching_resource_changed_totalCounter9.4Counter for ETag cache miss - resource changedendpoint
gitlab_transaction_event_fork_repository_totalCounter9.4Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists 
gitlab_transaction_event_import_repository_totalCounter9.4Counter for repository imports (RepositoryImportWorker) 
gitlab_transaction_event_patch_hard_limit_bytes_hit_totalCounter13.9Counter for diff patch size limit hits 
gitlab_transaction_event_push_branch_totalCounter9.4Counter for all branch pushes 
gitlab_transaction_event_rails_exception_totalCounter9.4Counter for number of rails exceptions 
gitlab_transaction_event_receive_email_totalCounter9.4Counter for received emailshandler
gitlab_transaction_event_remote_mirrors_failed_totalCounter10.8Counter for failed remote mirrors 
gitlab_transaction_event_remote_mirrors_finished_totalCounter10.8Counter for finished remote mirrors 
gitlab_transaction_event_remote_mirrors_running_totalCounter10.8Counter for running remote mirrors 
gitlab_transaction_event_remove_branch_totalCounter9.4Counter when a branch is removed for any repository 
gitlab_transaction_event_remove_repository_totalCounter9.4Counter when a repository is removed 
gitlab_transaction_event_remove_tag_totalCounter9.4Counter when a tag is remove for any repository 
gitlab_transaction_event_sidekiq_exception_totalCounter9.4Counter of Sidekiq exceptions 
gitlab_transaction_event_stuck_import_jobs_totalCounter9.4Count of stuck import jobs projects_without_jid_count, projects_with_jid_count
gitlab_transaction_event_update_build_totalCounter9.4Counter for update build for API /jobs/request/:id  
gitlab_transaction_new_redis_connections_totalCounter9.4Counter for new Redis connections 
gitlab_transaction_rails_queue_duration_totalCounter9.4Measures latency between GitLab Workhorse forwarding a request to Rails controller, action
gitlab_transaction_view_duration_totalCounter9.4Duration for views controller, action, view
gitlab_view_rendering_duration_secondsHistogram10.2Duration for views (histogram) controller, action, view
http_requests_totalCounter9.4Rack request count method, status
http_request_duration_secondsHistogram9.4HTTP response time from rack middleware for successful requestsmethod
gitlab_transaction_db_count_totalCounter13.1Counter for total number of SQL calls controller, action
gitlab_transaction_db_<role>_count_totalCounter13.10Counter for total number of SQL calls, grouped by database roles (primary/replica) controller, action
gitlab_transaction_db_write_count_totalCounter13.1Counter for total number of write SQL calls controller, action
gitlab_transaction_db_cached_count_totalCounter13.1Counter for total number of cached SQL calls controller, action
gitlab_transaction_db_<role>_cached_count_totalCounter13.1Counter for total number of cached SQL calls, grouped by database roles (primary/replica) controller, action
gitlab_transaction_db_<role>_wal_count_totalCounter14.0Counter for total number of WAL (write ahead log location) queries, grouped by database roles (primary/replica) controller, action
gitlab_transaction_db_<role>_wal_cached_count_totalCounter14.1Counter for total number of cached WAL (write ahead log location) queries, grouped by database roles (primary/replica) controller, action
http_elasticsearch_requests_duration_seconds Histogram13.1Elasticsearch requests duration during web transactions controller, action
http_elasticsearch_requests_total Counter13.1Elasticsearch requests count during web transactions controller, action
pipelines_created_totalCounter9.4Counter of pipelines created 
rack_uncaught_errors_totalCounter9.4Rack connections handling uncaught errors count 
user_session_logins_totalCounter9.4Counter of how many users have logged in since GitLab was started or restarted 
upload_file_does_not_existCounter10.7Number of times an upload record could not find its file. 
failed_login_captcha_totalGauge11.0Counter of failed CAPTCHA attempts during login 
successful_login_captcha_totalGauge11.0Counter of successful CAPTCHA attempts during login 
auto_devops_pipelines_completed_totalCounter12.7Counter of completed Auto DevOps pipelines, labeled by status 
artifact_report_<report_type>_builds_completed_totalCounter15.3Counter of completed CI Builds with report-type artifacts, grouped by report type and labeled by status 
gitlab_metrics_dashboard_processing_time_msSummary12.10Metrics dashboard processing time in millisecondsservice, stages
action_cable_active_connectionsGauge13.4Number of ActionCable WS clients currently connectedserver_mode
action_cable_broadcasts_totalCounter13.10The number of ActionCable broadcasts emittedserver_mode
action_cable_pool_min_sizeGauge13.4Minimum number of worker threads in ActionCable thread poolserver_mode
action_cable_pool_max_sizeGauge13.4Maximum number of worker threads in ActionCable thread poolserver_mode
action_cable_pool_current_sizeGauge13.4Current number of worker threads in ActionCable thread poolserver_mode
action_cable_pool_largest_sizeGauge13.4Largest number of worker threads observed so far in ActionCable thread poolserver_mode
action_cable_pool_pending_tasksGauge13.4Number of tasks waiting to be executed in ActionCable thread poolserver_mode
action_cable_pool_tasks_totalGauge13.4Total number of tasks executed in ActionCable thread poolserver_mode
gitlab_ci_trace_operations_totalCounter13.4Total amount of different operations on a build traceoperation
gitlab_ci_trace_bytes_totalCounter13.4Total amount of build trace bytes transferred 
action_cable_single_client_transmissions_totalCounter13.10The number of ActionCable messages transmitted to any client in any channelserver_mode
action_cable_subscription_confirmations_totalCounter13.10The number of ActionCable subscriptions from clients confirmedserver_mode
action_cable_subscription_rejections_totalCounter13.10The number of ActionCable subscriptions from clients rejectedserver_mode
action_cable_transmitted_bytes_totalCounter16.0Total number of bytes transmitted over ActionCable operation, channel
gitlab_issuable_fast_count_by_state_totalCounter13.5Total number of row count operations on issue/merge request list pages 
gitlab_issuable_fast_count_by_state_failures_totalCounter13.5Number of soft-failed row count operations on issue/merge request list pages 
gitlab_ci_trace_finalize_duration_secondsHistogram13.6Duration of build trace chunks migration to object storage 
gitlab_vulnerability_report_branch_comparison_real_duration_secondsHistogram15.11Execution duration of vulnerability report present on default branch sql query 
gitlab_vulnerability_report_branch_comparison_cpu_duration_secondsHistogram15.11Execution duration of vulnerability report present on default branch sql query 
gitlab_external_http_totalCounter13.8Total number of HTTP calls to external systems controller, action
gitlab_external_http_duration_secondsCounter13.8Duration in seconds spent on each HTTP call to external systems 
gitlab_external_http_exception_totalCounter13.8Total number of exceptions raised when making external HTTP calls 
ci_report_parser_duration_secondsHistogram13.9Time to parse CI/CD report artifactsparser
pipeline_graph_link_calculation_duration_secondsHistogram13.9Total time spent calculating links, in seconds 
pipeline_graph_links_totalHistogram13.9Number of links per graph 
pipeline_graph_links_per_job_ratioHistogram13.9Ratio of links to job per graph 
gitlab_ci_pipeline_security_orchestration_policy_processing_duration_secondsHistogram13.12Time in seconds it takes to process Security Policies in CI/CD pipeline 
gitlab_spamcheck_request_duration_secondsHistogram13.12The duration for requests between Rails and the anti-spam engine 
service_desk_thank_you_emailCounter14.0Total number of email responses to new Service Desk emails 
service_desk_new_note_emailCounter14.0Total number of email notifications on new Service Desk comment 
email_receiver_errorCounter14.1Total number of errors when processing incoming emails 
gitlab_snowplow_events_totalCounter14.1Total number of GitLab Snowplow Analytics Instrumentation events emitted 
gitlab_snowplow_failed_events_totalCounter14.1Total number of GitLab Snowplow Analytics Instrumentation events emission failures 
gitlab_snowplow_successful_events_totalCounter14.1Total number of GitLab Snowplow Analytics Instrumentation events emission successes 
gitlab_ci_build_trace_errors_totalCounter14.4Total amount of different error types on a build traceerror_reason
gitlab_presentable_object_cacheless_render_real_duration_secondsHistogram15.3Duration of real time spent caching and representing specific web request objects controller, action
cached_object_operations_totalCounter15.3Total number of objects cached for specific web requests controller, action
redis_hit_miss_operations_totalCounter15.6Total number of Redis cache hits and misses cache_hit, cache_identifier, feature_category, backing_resource
redis_cache_generation_duration_secondsHistogram15.6Time to generate Redis cache cache_hit, cache_identifier, feature_category, backing_resource
gitlab_diffs_reorder_real_duration_secondsHistogram15.8Duration in seconds spend on reordering of diff files on diffs batch request controller, action
gitlab_diffs_collection_real_duration_secondsHistogram15.8Duration in seconds spent on querying merge request diff files on diffs batch request controller, action
gitlab_diffs_comparison_real_duration_secondsHistogram15.8Duration in seconds spent on getting comparison data on diffs batch request controller, action
gitlab_diffs_unfoldable_positions_real_duration_secondsHistogram15.8Duration in seconds spent on getting unfoldable note positions on diffs batch request controller, action
gitlab_diffs_unfold_real_duration_secondsHistogram15.8Duration in seconds spent on unfolding positions on diffs batch request controller, action
gitlab_diffs_write_cache_real_duration_secondsHistogram15.8Duration in seconds spent on caching highlighted lines and stats on diffs batch request controller, action
gitlab_diffs_highlight_cache_decorate_real_duration_secondsHistogram15.8Duration in seconds spent on setting highlighted lines from cache on diffs batch request controller, action
gitlab_diffs_render_real_duration_secondsHistogram15.8Duration in seconds spent on serializing and rendering diffs on diffs batch request controller, action
gitlab_memwd_violations_totalCounter15.9Total number of times a Ruby process violated a memory threshold 
gitlab_memwd_violations_handled_totalCounter15.9Total number of times Ruby process memory violations were handled 
gitlab_sli_rails_request_apdex_totalCounter14.4Total number of request Apdex measurements. For more information, see Rails request SLIs endpoint_id, feature_category, request_urgency
gitlab_sli_rails_request_apdex_success_totalCounter14.4Total number of successful requests that met the target duration for their urgency. Divide by gitlab_sli_rails_requests_apdex_total to get a success ratio endpoint_id, feature_category, request_urgency
gitlab_sli_rails_request_error_totalCounter15.7Total number of request error measurements. For more information, see Rails request SLIs endpoint_id, feature_category, request_urgency, error
job_register_attempts_failed_totalCounter9.5Counts the times a runner fails to register a job 
job_register_attempts_totalCounter9.5Counts the times a runner tries to register a job 
job_queue_duration_secondsHistogram9.5Request handling execution time 
gitlab_ci_queue_operations_totalCounter16.3Counts all the operations that are happening inside a queue 
gitlab_ci_queue_depth_totalHistogram16.3Size of a CI/CD builds queue in relation to the operation result 
gitlab_ci_queue_size_totalHistogram16.3Size of initialized CI/CD builds queue 
gitlab_ci_current_queue_sizeGauge16.3Current size of initialized CI/CD builds queue 
gitlab_ci_queue_iteration_duration_secondsHistogram16.3Time it takes to find a build in CI/CD queue 
gitlab_ci_queue_retrieval_duration_secondsHistogram16.3Time it takes to execute a SQL query to retrieve builds queue 
gitlab_ci_queue_active_runners_totalHistogram16.3The amount of active runners that can process queue in a project 

Metrics controlled by a feature flag

The following metrics can be controlled by feature flags:

MetricFeature flag
gitlab_view_rendering_duration_secondsprometheus_metrics_view_instrumentation
gitlab_ci_queue_depth_totalgitlab_ci_builds_queuing_metrics
gitlab_ci_queue_sizegitlab_ci_builds_queuing_metrics
gitlab_ci_queue_size_totalgitlab_ci_builds_queuing_metrics
gitlab_ci_queue_iteration_duration_secondsgitlab_ci_builds_queuing_metrics
gitlab_ci_current_queue_sizegitlab_ci_builds_queuing_metrics
gitlab_ci_queue_retrieval_duration_secondsgitlab_ci_builds_queuing_metrics
gitlab_ci_queue_active_runners_totalgitlab_ci_builds_queuing_metrics

Praefect metrics

You can configure Praefect to report metrics. For information on available metrics, see the relevant documentation.

Sidekiq metrics

Sidekiq jobs may also gather metrics, and these metrics can be accessed if the Sidekiq exporter is enabled: for example, using the monitoring.sidekiq_exporter configuration option in gitlab.yml. These metrics are served from the /metrics path on the configured port.

MetricTypeSinceDescriptionLabels
sidekiq_jobs_cpu_secondsHistogram12.4Seconds of CPU time to run Sidekiq job queue, boundary, external_dependencies, feature_category, job_status, urgency
sidekiq_jobs_completion_secondsHistogram12.2Seconds to complete Sidekiq job queue, boundary, external_dependencies, feature_category, job_status, urgency
sidekiq_jobs_db_secondsHistogram12.9Seconds of DB time to run Sidekiq job queue, boundary, external_dependencies, feature_category, job_status, urgency
sidekiq_jobs_gitaly_secondsHistogram12.9Seconds of Gitaly time to run Sidekiq job queue, boundary, external_dependencies, feature_category, job_status, urgency
sidekiq_redis_requests_duration_secondsHistogram13.1Duration in seconds that a Sidekiq job spent querying a Redis server queue, boundary, external_dependencies, feature_category, job_status, urgency
sidekiq_elasticsearch_requests_duration_secondsHistogram13.1Duration in seconds that a Sidekiq job spent in requests to an Elasticsearch server queue, boundary, external_dependencies, feature_category, job_status, urgency
sidekiq_jobs_queue_duration_secondsHistogram12.5Duration in seconds that a Sidekiq job was queued before being executed queue, boundary, external_dependencies, feature_category, urgency
sidekiq_jobs_failed_totalCounter12.2Sidekiq jobs failed queue, boundary, external_dependencies, feature_category, urgency
sidekiq_jobs_retried_totalCounter12.2Sidekiq jobs retried queue, boundary, external_dependencies, feature_category, urgency
sidekiq_jobs_interrupted_totalCounter15.2Sidekiq jobs interrupted queue, boundary, external_dependencies, feature_category, urgency
sidekiq_jobs_dead_totalCounter13.7Sidekiq dead jobs (jobs that have run out of retries) queue, boundary, external_dependencies, feature_category, urgency
sidekiq_redis_requests_totalCounter13.1Redis requests during a Sidekiq job execution queue, boundary, external_dependencies, feature_category, job_status, urgency
sidekiq_elasticsearch_requests_totalCounter13.1Elasticsearch requests during a Sidekiq job execution queue, boundary, external_dependencies, feature_category, job_status, urgency
sidekiq_jobs_skipped_totalCounter16.2Number of jobs being skipped (dropped or deferred) when drop_sidekiq_jobs feature flag is enabled or run_sidekiq_jobs feature flag is disabled worker, action
sidekiq_running_jobsGauge12.2Number of Sidekiq jobs running queue, boundary, external_dependencies, feature_category, urgency
sidekiq_concurrencyGauge12.5Maximum number of Sidekiq jobs 
sidekiq_mem_total_bytesGauge15.3Number of bytes allocated for both objects consuming an object slot and objects that required a malloc’ 
geo_db_replication_lag_secondsGauge10.2Database replication lag (seconds)url
geo_repositoriesGauge10.2Total number of repositories available on primaryurl
geo_repositories_syncedGauge10.2Number of repositories synced on secondaryurl
geo_repositories_failedGauge10.2Number of repositories failed to sync on secondaryurl
geo_lfs_objectsGauge10.2Number of LFS objects on primaryurl
geo_lfs_objects_checksummedGauge14.6Number of LFS objects checksummed successfully on primaryurl
geo_lfs_objects_checksum_failedGauge14.6Number of LFS objects failed to calculate the checksum on primaryurl
geo_lfs_objects_checksum_totalGauge14.6Number of LFS objects that need to be checksummed on primaryurl
geo_lfs_objects_syncedGauge10.2Number of syncable LFS objects synced on secondaryurl
geo_lfs_objects_failedGauge10.2Number of syncable LFS objects failed to sync on secondaryurl
geo_lfs_objects_registryGauge14.6Number of LFS objects in the registryurl
geo_lfs_objects_verifiedGauge14.6Number of LFS objects successfully verified on secondaryurl
geo_lfs_objects_verification_failedGauge14.6Number of LFS objects that failed verifications on secondaryurl
geo_lfs_objects_verification_totalGauge14.6Number of LFS objects to attempt to verify on secondaryurl
geo_last_event_idGauge10.2Database ID of the latest event log entry on the primaryurl
geo_last_event_timestampGauge10.2UNIX timestamp of the latest event log entry on the primaryurl
geo_cursor_last_event_idGauge10.2Last database ID of the event log processed by the secondaryurl
geo_cursor_last_event_timestampGauge10.2Last UNIX timestamp of the event log processed by the secondaryurl
geo_status_failed_totalCounter10.2Number of times retrieving the status from the Geo Node failedurl
geo_last_successful_status_check_timestampGauge10.2Last timestamp when the status was successfully updatedurl
geo_job_artifacts_synced_missing_on_primaryGauge10.7Number of job artifacts marked as synced due to the file missing on the primaryurl
geo_repositories_checksummedGauge10.7Number of repositories checksummed on primaryurl
geo_repositories_checksum_failedGauge10.7Number of repositories failed to calculate the checksum on primaryurl
geo_repositories_verifiedGauge10.7Number of repositories successfully verified on secondaryurl
geo_repositories_verification_failedGauge10.7Number of repositories that failed verification on secondaryurl
geo_repositories_checksum_mismatchGauge10.7Number of repositories that checksum mismatch on secondaryurl
geo_repositories_checkedGauge11.1Number of repositories that have been checked via git fsck url
geo_repositories_checked_failedGauge11.1Number of repositories that have a failure from git fsck url
geo_repositories_retrying_verificationGauge11.2Number of repositories verification failures that Geo is actively trying to correct on secondaryurl
geo_package_filesGauge13.0Number of package files on primaryurl
geo_package_files_checksummedGauge13.0Number of package files checksummed on primaryurl
geo_package_files_checksum_failedGauge13.0Number of package files failed to calculate the checksum on primaryurl
geo_package_files_syncedGauge13.3Number of syncable package files synced on secondaryurl
geo_package_files_failedGauge13.3Number of syncable package files failed to sync on secondaryurl
geo_package_files_registryGauge13.3Number of package files in the registryurl
geo_terraform_state_versionsGauge13.5Number of terraform state versions on primaryurl
geo_terraform_state_versions_checksummedGauge13.5Number of terraform state versions checksummed successfully on primaryurl
geo_terraform_state_versions_checksum_failedGauge13.5Number of terraform state versions failed to calculate the checksum on primaryurl
geo_terraform_state_versions_checksum_totalGauge13.12Number of terraform state versions that need to be checksummed on primaryurl
geo_terraform_state_versions_syncedGauge13.5Number of syncable terraform state versions synced on secondaryurl
geo_terraform_state_versions_failedGauge13.5Number of syncable terraform state versions failed to sync on secondaryurl
geo_terraform_state_versions_registryGauge13.5Number of terraform state versions in the registryurl
geo_terraform_state_versions_verifiedGauge13.12Number of terraform state versions successfully verified on secondaryurl
geo_terraform_state_versions_verification_failedGauge13.12Number of terraform state versions that failed verification on secondaryurl
geo_terraform_state_versions_verification_totalGauge13.12Number of terraform state versions to attempt to verify on secondaryurl
global_search_bulk_cron_queue_sizeGauge12.10Number of database records waiting to be synchronized to Elasticsearch 
global_search_awaiting_indexing_queue_sizeGauge13.2Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused 
geo_merge_request_diffsGauge13.4Number of merge request diffs on primaryurl
geo_merge_request_diffs_checksum_totalGauge13.12Number of merge request diffs to checksum on primaryurl
geo_merge_request_diffs_checksummedGauge13.4Number of merge request diffs that successfully calculated the checksum on primaryurl
geo_merge_request_diffs_checksum_failedGauge13.4Number of merge request diffs failed to calculate the checksum on primaryurl
geo_merge_request_diffs_syncedGauge13.4Number of syncable merge request diffs synced on secondaryurl
geo_merge_request_diffs_failedGauge13.4Number of syncable merge request diffs failed to sync on secondaryurl
geo_merge_request_diffs_registryGauge13.4Number of merge request diffs in the registryurl
geo_merge_request_diffs_verification_totalGauge13.12Number of merge request diffs to attempt to verify on secondaryurl
geo_merge_request_diffs_verifiedGauge13.12Number of merge request diffs successfully verified on secondaryurl
geo_merge_request_diffs_verification_failedGauge13.12Number of merge request diffs that failed verification on secondaryurl
geo_snippet_repositoriesGauge13.4Number of snippets on primaryurl
geo_snippet_repositories_checksummedGauge13.4Number of snippets checksummed on primaryurl
geo_snippet_repositories_checksum_failedGauge13.4Number of snippets failed to calculate the checksum on primaryurl
geo_snippet_repositories_syncedGauge13.4Number of syncable snippets synced on secondaryurl
geo_snippet_repositories_failedGauge13.4Number of syncable snippets failed on secondaryurl
geo_snippet_repositories_registryGauge13.4Number of syncable snippets in the registryurl
geo_group_wiki_repositoriesGauge13.10Number of group wikis on primaryurl
geo_group_wiki_repositories_checksum_totalGauge16.3Number of group wikis to checksum on primaryurl
geo_group_wiki_repositories_checksummedGauge13.10Number of group wikis that successfully calculated the checksum on primaryurl
geo_group_wiki_repositories_checksum_failedGauge13.10Number of group wikis that failed to calculate the checksum on primaryurl
geo_group_wiki_repositories_syncedGauge13.10Number of syncable group wikis synced on secondaryurl
geo_group_wiki_repositories_failedGauge13.10Number of syncable group wikis failed to sync on secondaryurl
geo_group_wiki_repositories_registryGauge13.10Number of group wikis in the registryurl
geo_group_wiki_repositories_verification_totalGauge16.3Number of group wikis to attempt to verify on secondaryurl
geo_group_wiki_repositories_verifiedGauge16.3Number of group wikis successfully verified on secondaryurl
geo_group_wiki_repositories_verification_failedGauge16.3Number of group wikis that failed verification on secondaryurl
geo_pages_deploymentsGauge14.3Number of pages deployments on primaryurl
geo_pages_deployments_checksum_totalGauge14.6Number of pages deployments to checksum on primaryurl
geo_pages_deployments_checksummedGauge14.6Number of pages deployments that successfully calculated the checksum on primaryurl
geo_pages_deployments_checksum_failedGauge14.6Number of pages deployments failed to calculate the checksum on primaryurl
geo_pages_deployments_syncedGauge14.3Number of syncable pages deployments synced on secondaryurl
geo_pages_deployments_failedGauge14.3Number of syncable pages deployments failed to sync on secondaryurl
geo_pages_deployments_registryGauge14.3Number of pages deployments in the registryurl
geo_pages_deployments_verification_totalGauge14.6Number of pages deployments to attempt to verify on secondaryurl
geo_pages_deployments_verifiedGauge14.6Number of pages deployments successfully verified on secondaryurl
geo_pages_deployments_verification_failedGauge14.6Number of pages deployments verifications failed on secondaryurl
geo_job_artifactsGauge14.8Number of job artifacts on primaryurl
geo_job_artifacts_checksum_totalGauge14.8Number of job artifacts to checksum on primaryurl
geo_job_artifacts_checksummedGauge14.8Number of job artifacts that successfully calculated the checksum on primaryurl
geo_job_artifacts_checksum_failedGauge14.8Number of job artifacts failed to calculate the checksum on primaryurl
geo_job_artifacts_syncedGauge14.8Number of syncable job artifacts synced on secondaryurl
geo_job_artifacts_failedGauge14.8Number of syncable job artifacts failed to sync on secondaryurl
geo_job_artifacts_registryGauge14.8Number of job artifacts in the registryurl
geo_job_artifacts_verification_totalGauge14.8Number of job artifacts to attempt to verify on secondaryurl
geo_job_artifacts_verifiedGauge14.8Number of job artifacts successfully verified on secondaryurl
geo_job_artifacts_verification_failedGauge14.8Number of job artifacts that failed verification on secondaryurl
limited_capacity_worker_running_jobsGauge13.5Number of running jobsworker
limited_capacity_worker_max_running_jobsGauge13.5Maximum number of running jobsworker
limited_capacity_worker_remaining_work_countGauge13.5Number of jobs waiting to be enqueuedworker
destroyed_job_artifacts_count_totalCounter13.6Number of destroyed expired job artifacts 
destroyed_pipeline_artifacts_count_totalCounter13.8Number of destroyed expired pipeline artifacts 
gitlab_optimistic_locking_retriesHistogram13.10Number of retry attempts to execute optimistic retry lock 
geo_uploadsGauge14.1Number of uploads on primaryurl
geo_uploads_syncedGauge14.1Number of uploads synced on secondaryurl
geo_uploads_failedGauge14.1Number of syncable uploads failed to sync on secondaryurl
geo_uploads_registryGauge14.1Number of uploads in the registryurl
geo_uploads_checksum_totalGauge14.6Number of uploads to checksum on primaryurl
geo_uploads_checksummedGauge14.6Number of uploads that successfully calculated the checksum on primaryurl
geo_uploads_checksum_failedGauge14.6Number of uploads failed to calculate the checksum on primaryurl
geo_uploads_verification_totalGauge14.6Number of uploads to attempt to verify on secondaryurl
geo_uploads_verifiedGauge14.6Number of uploads successfully verified on secondaryurl
geo_uploads_verification_failedGauge14.6Number of uploads that failed verification on secondaryurl
geo_container_repositoriesGauge15.4Number of container repositories on primaryurl
geo_container_repositories_syncedGauge15.4Number of container repositories synced on secondaryurl
geo_container_repositories_failedGauge15.4Number of syncable container repositories failed to sync on secondaryurl
geo_container_repositories_registryGauge15.4Number of container repositories in the registryurl
geo_container_repositories_checksum_totalGauge15.10Number of container repositories checksummed successfully on primaryurl
geo_container_repositories_checksummedGauge15.10Number of container repositories tried to checksum on primaryurl
geo_container_repositories_checksum_failedGauge15.10Number of container repositories failed to calculate the checksum on primaryurl
geo_container_repositories_verification_totalGauge15.10Number of container repositories’ verifications tried on secondaryurl
geo_container_repositories_verifiedGauge15.10Number of container repositories verified on secondaryurl
geo_container_repositories_verification_failedGauge15.10Number of container repositories’ failed verifications on secondaryurl
geo_ci_secure_filesGauge15.3Number of secure files on primaryurl
geo_ci_secure_files_checksum_totalGauge15.3Number of secure files to checksum on primaryurl
geo_ci_secure_files_checksummedGauge15.3Number of secure files that successfully calculated the checksum on primaryurl
geo_ci_secure_files_checksum_failedGauge15.3Number of secure files failed to calculate the checksum on primaryurl
geo_ci_secure_files_syncedGauge15.3Number of syncable secure files synced on secondaryurl
geo_ci_secure_files_failedGauge15.3Number of syncable secure files failed to sync on secondaryurl
geo_ci_secure_files_registryGauge15.3Number of secure files in the registryurl
geo_ci_secure_files_verification_totalGauge15.3Number of secure files to attempt to verify on secondaryurl
geo_ci_secure_files_verifiedGauge15.3Number of secure files successfully verified on secondaryurl
geo_ci_secure_files_verification_failedGauge15.3Number of secure files that failed verification on secondaryurl
geo_dependency_proxy_blobGauge15.6Number of dependency proxy blobs on primary 
geo_dependency_proxy_blob_checksum_totalGauge15.6Number of dependency proxy blobs to checksum on primary 
geo_dependency_proxy_blob_checksummedGauge15.6Number of dependency proxy blobs that successfully calculated the checksum on primary 
geo_dependency_proxy_blob_checksum_failedGauge15.6Number of dependency proxy blobs failed to calculate the checksum on primary 
geo_dependency_proxy_blob_syncedGauge15.6Number of dependency proxy blobs synced on secondary 
geo_dependency_proxy_blob_failedGauge15.6Number of dependency proxy blobs failed to sync on secondary 
geo_dependency_proxy_blob_registryGauge15.6Number of dependency proxy blobs in the registry 
geo_dependency_proxy_blob_verification_totalGauge15.6Number of dependency proxy blobs to attempt to verify on secondary 
geo_dependency_proxy_blob_verifiedGauge15.6Number of dependency proxy blobs successfully verified on secondary 
geo_dependency_proxy_blob_verification_failedGauge15.6Number of dependency proxy blobs that failed verification on secondary 
geo_dependency_proxy_manifestsGauge15.6Number of dependency proxy manifests on primaryurl
geo_dependency_proxy_manifests_checksum_totalGauge15.6Number of dependency proxy manifests to checksum on primaryurl
geo_dependency_proxy_manifests_checksummedGauge15.6Number of dependency proxy manifests that successfully calculated the checksum on primaryurl
geo_dependency_proxy_manifests_checksum_failedGauge15.6Number of dependency proxy manifests failed to calculate the checksum on primaryurl
geo_dependency_proxy_manifests_syncedGauge15.6Number of syncable dependency proxy manifests synced on secondaryurl
geo_dependency_proxy_manifests_failedGauge15.6Number of syncable dependency proxy manifests failed to sync on secondaryurl
geo_dependency_proxy_manifests_registryGauge15.6Number of dependency proxy manifests in the registryurl
geo_dependency_proxy_manifests_verification_totalGauge15.6Number of dependency proxy manifests to attempt to verify on secondaryurl
geo_dependency_proxy_manifests_verifiedGauge15.6Number of dependency proxy manifests successfully verified on secondaryurl
geo_dependency_proxy_manifests_verification_failedGauge15.6Number of dependency proxy manifests that failed verification on secondaryurl
geo_project_wiki_repositoriesGauge15.10Number of Project Wiki Repositories on primaryurl
geo_project_wiki_repositories_checksum_totalGauge15.10Number of Project Wiki Repositories to checksum on primaryurl
geo_project_wiki_repositories_checksummedGauge15.10Number of Project Wiki Repositories that successfully calculated the checksum on primaryurl
geo_project_wiki_repositories_checksum_failedGauge15.10Number of Project Wiki Repositories that failed to calculate the checksum on primaryurl
geo_project_wiki_repositories_syncedGauge15.10Number of syncable Project Wiki Repositories synced on secondaryurl
geo_project_wiki_repositories_failedGauge15.10Number of syncable Project Wiki Repositories failed to sync on secondaryurl
geo_project_wiki_repositories_registryGauge15.10Number of Project Wiki Repositories in the registryurl
geo_project_wiki_repositories_verification_totalGauge15.10Number of Project Wiki Repositories to attempt to verify on secondaryurl
geo_project_wiki_repositories_verifiedGauge15.10Number of Project Wiki Repositories successfully verified on secondaryurl
geo_project_wiki_repositories_verification_failedGauge15.10Number of Project Wiki Repositories that failed verification on secondaryurl
geo_project_repositoriesGauge16.2Number of Project Repositories on primaryurl
geo_project_repositories_checksum_totalGauge16.2Number of Project Repositories to checksum on primaryurl
geo_project_repositories_checksummedGauge16.2Number of Project Repositories that successfully calculated the checksum on primaryurl
geo_project_repositories_checksum_failedGauge16.2Number of Project Repositories that failed to calculate the checksum on primaryurl
geo_project_repositories_syncedGauge16.2Number of syncable Project Repositories synced on secondaryurl
geo_project_repositories_failedGauge16.2Number of syncable Project Repositories failed to sync on secondaryurl
geo_project_repositories_registryGauge16.2Number of Project Repositories in the registryurl
geo_project_repositories_verification_totalGauge16.2Number of Project Repositories to attempt to verify on secondaryurl
geo_project_repositories_verifiedGauge16.2Number of Project Repositories successfully verified on secondaryurl
geo_project_repositories_verification_failedGauge16.2Number of Project Repositories that failed verification on secondaryurl
gitlab_memwd_violations_totalCounter15.9Total number of times a Sidekiq process violated a memory threshold 
gitlab_memwd_violations_handled_totalCounter15.9Total number of times Sidekiq process memory violations were handled 
sidekiq_watchdog_running_jobs_totalCounter15.9Current running jobs when RSS limit was reachedworker_class
gitlab_maintenance_modeGauge15.11Is GitLab Maintenance Mode enabled? 
geo_design_management_repositoriesGauge16.1Number of design repositories on primaryurl
geo_design_management_repositories_checksum_totalGauge16.1Number of design repositories tried to checksum on primaryurl
geo_design_management_repositories_checksummedGauge16.1Number of design repositories successfully checksummed on primaryurl
geo_design_management_repositories_checksum_failedGauge16.1Number of design repositories failed to calculate the checksum on primaryurl
geo_design_management_repositories_syncedGauge16.1Number of syncable design repositories synced on secondaryurl
geo_design_management_repositories_failedGauge16.1Number of syncable design repositories failed to sync on secondaryurl
geo_design_management_repositories_registryGauge16.1Number of design repositories in the registryurl
geo_design_management_repositories_verification_totalGauge16.1Number of design repositories verifications tried on secondaryurl
geo_design_management_repositories_verifiedGauge16.1Number of design repositories verified on secondaryurl
geo_design_management_repositories_verification_failedGauge16.1Number of design repositories verifications failed on secondaryurl

Database load balancing metrics

The following metrics are available:

MetricTypeSinceDescriptionLabels
db_load_balancing_hostsGauge12.3Current number of load balancing hosts 
sidekiq_load_balancing_countCounter13.11Sidekiq jobs using load balancing with data consistency set to :sticky or :delayed queue, boundary, external_dependencies, feature_category, job_status, urgency, data_consistency, load_balancing_strategy
gitlab_transaction_caught_up_replica_pick_count_totalCounter14.1Number of search attempts for caught up replicaresult

Database partitioning metrics

The following metrics are available:

MetricTypeSinceDescription
db_partitions_presentGauge13.4Number of database partitions present
db_partitions_missingGauge13.4Number of database partitions currently expected, but not present

Connection pool metrics

These metrics record the status of the database connection pools, and the metrics all have these labels:

  • class - the Ruby class being recorded.
    • ActiveRecord::Base is the main database connection.
    • Geo::TrackingBase is the connection to the Geo tracking database, if enabled.
  • host - the host name used to connect to the database.
  • port - the port used to connect to the database.
MetricTypeSinceDescription
gitlab_database_connection_pool_sizeGauge13.0Total connection pool capacity
gitlab_database_connection_pool_connectionsGauge13.0Current connections in the pool
gitlab_database_connection_pool_busyGauge13.0Connections in use where the owner is still alive
gitlab_database_connection_pool_deadGauge13.0Connections in use where the owner is not alive
gitlab_database_connection_pool_idleGauge13.0Connections not in use
gitlab_database_connection_pool_waitingGauge13.0Threads currently waiting on this queue

Ruby metrics

Some basic Ruby runtime metrics are available:

MetricTypeSinceDescription
ruby_gc_duration_secondsCounter11.1Time spent by Ruby in GC
ruby_gc_stat_...Gauge11.1Various metrics from GC.stat
ruby_gc_stat_ext_heap_fragmentationGauge15.2Degree of Ruby heap fragmentation as live objects versus eden slots (range 0 to 1)
ruby_file_descriptorsGauge11.1File descriptors per process
ruby_sampler_duration_secondsCounter11.1Time spent collecting stats
ruby_process_cpu_seconds_totalGauge12.0Total amount of CPU time per process
ruby_process_max_fdsGauge12.0Maximum number of open file descriptors per process
ruby_process_resident_memory_bytesGauge12.0Memory usage by process (RSS/Resident Set Size)
ruby_process_resident_anon_memory_bytesGauge15.6Anonymous memory usage by process (RSS/Resident Set Size)
ruby_process_resident_file_memory_bytesGauge15.6File-backed memory usage by process (RSS/Resident Set Size)
ruby_process_unique_memory_bytesGauge13.0Memory usage by process (USS/Unique Set Size)
ruby_process_proportional_memory_bytesGauge13.0Memory usage by process (PSS/Proportional Set Size)
ruby_process_start_time_secondsGauge12.0UNIX timestamp of process start time

Puma Metrics

MetricTypeSinceDescription
puma_workersGauge12.0Total number of workers
puma_running_workersGauge12.0Number of booted workers
puma_stale_workersGauge12.0Number of old workers
puma_runningGauge12.0Number of running threads
puma_queued_connectionsGauge12.0Number of connections in that worker’s “to do” set waiting for a worker thread
puma_active_connectionsGauge12.0Number of threads processing a request
puma_pool_capacityGauge12.0Number of requests the worker is capable of taking right now
puma_max_threadsGauge12.0Maximum number of worker threads
puma_idle_threadsGauge12.0Number of spawned threads which are not processing a request

Redis metrics

These client metrics are meant to complement Redis server metrics. These metrics are broken down per Redis instance. These metrics all have a storage label which indicates the Redis instance. For example, cache or shared_state.

MetricTypeSinceDescription
gitlab_redis_client_exceptions_totalCounter13.2Number of Redis client exceptions, broken down by exception class
gitlab_redis_client_requests_totalCounter13.2Number of Redis client requests
gitlab_redis_client_requests_duration_secondsHistogram13.2Redis request latency, excluding blocking commands
gitlab_redis_client_redirections_totalCounter15.10Number of Redis Cluster MOVED/ASK redirections, broken down by redirection type
gitlab_redis_client_requests_pipelined_commandsHistogram16.4Number of commands per pipeline sent to a single Redis server

Metrics shared directory

The GitLab Prometheus client requires a directory to store metrics data shared between multi-process services. Those files are shared among all instances running under Puma server. The directory must be accessible to all running Puma’s processes, or metrics can’t function correctly.

This directory’s location is configured using environment variable prometheus_multiproc_dir. For best performance, create this directory in tmpfs.

If GitLab is installed using the Linux package and tmpfs is available, then GitLab configures the metrics directory for you.