Metrics for measuring monorepo performance
The following metrics can be used to measure server side performance of your monorepo. These metrics are not limited to monorepo performance and are more general metrics to measure Gitaly performance, but they are especially relevant when running a monorepo.
Clones and Fetches
The most frequent expensive operation are clones and fetches. When taken as a percentage of system resources consumed, these operations often contribute to 90% or more of system resources on Gitaly nodes. Here are some logs and metrics that can provide useful signals.
CPU and Memory
There are two main RPCs that handle clones/fetches. The following log entry fields an be used to inspect how much system resources are consumed by clones/fetches for a given repository.
The following are log entry fields in the Gitaly logs that can be filtered on:
| Log field | Values to filter on | Why? |
|---|---|---|
json.grpc.method | PostReceivePack | This is the RPC that handles HTTP clones/fetches |
json.grpc.method | SSHReceivePack | This is the RPC that handles SSH clones/fetches |
json.grpc.code | OK | Indicates the RPC has successfully served its request |
json.grpc.code | Canceled | Often times indicates the client killed the connection, usually due to a timeout of some sort |
json.grpc.code | ResourceExhausted | Indicates there are too many Git processes being spawned on the machine simultaneously |
json.user_id | A user_id who initiated the clone/fetch. This is in the form of user-<user_id> eg: user-22345
| Indicates there are too many Git processes being spawned on the machine simultaneously |
json.username | A username who initiated the clone/fetch. eg: ilovecoding
| In order to see how many clones/fetches were from a given user. This is sometimes helpful to find excessive clone operations by a single user |
json.grpc.request.glRepository | A repository in question. In the form of project-<project_id> eg: project-214
| In order to see how many clones/fetches were for a given repository. |
json.grpc.request.glProjectPath | A repository in question. In the form of a project path eg: my-org/coolproject
| In order to see how many clones/fetches were for a given repository. |
The following are log entry fields that give useful information about cpu and memory:
| Log field to inspect | What does it tell you? |
|---|---|
json.command.cpu_time_ms | How much CPU time used by subprocesses this RPC spawned |
json.command.maxrss | How much memory was consumed from subprocesses this RPC spawned |
Example log message:
{
"command.count":2,
"command.cpu_time_ms":420,
"command.inblock":0,
"command.majflt":0,
"command.maxrss":3342152,
"command.minflt":24316,
"command.oublock":56,
"command.real_time_ms":626,
"command.spawn_token_fork_ms":4,
"command.spawn_token_wait_ms":0,
"command.system_time_ms":172,
"command.user_time_ms":248,
"component":"gitaly.StreamServerInterceptor",
"correlation_id":"20HCB3DAEPLV08UGNIYT9HJ4JW",
"environment":"gprd",
"feature_flags":"",
"fqdn":"file-99-stor-gprd.c.gitlab-production.internal",
"grpc.code":"OK",
"grpc.meta.auth_version":"v2",
"grpc.meta.client_name":"gitlab-workhorse",
"grpc.meta.deadline_type":"none",
"grpc.meta.method_operation":"mutator",
"grpc.meta.method_scope":"repository",
"grpc.meta.method_type":"bidi_stream",
"grpc.method":"PostReceivePack",
"grpc.request.fullMethod":"/gitaly.SmartHTTPService/PostReceivePack",
"grpc.request.glProjectPath":"r2414/revenir/development/machinelearning/protein-ddg",
"grpc.request.glRepository":"project-47506374",
"grpc.request.payload_bytes":911,
"grpc.request.repoPath":"@hashed/db/ab/dbabf83f57affedc9a001dc6c6f6b47bb431bd47d7254edd1daf24f0c38793a9.git",
"grpc.request.repoStorage":"nfs-file99",
"grpc.response.payload_bytes":54
"grpc.service":"gitaly.SmartHTTPService",
"grpc.start_time":"2023-10-16T20:40:08.836",
"grpc.time_ms":631.486,
"hostname":"file-99-stor-gprd",
"level":"info",
"msg":"finished streaming call with code OK",
"pid":1741362,
"remote_ip":"108.163.136.48",
"shard":"default",
"span.kind":"server",
"stage":"main",
"system":"grpc",
"tag":"gitaly",
"tier":"stor",
"time":"2023-10-16T20:40:09.467Z",
"trace.traceid":"AAB3QAeD8G+H9VNmzOi2CztMAcJv1+g4+l1cAgA=",
"type":"gitaly",
"user_id":"user-14857500",
"username":"ctx_ckottke",
}
Read distribution
The gitaly_praefect_read_distribution Prometheus metric is a
counter that
indicates how many reads have gone to which Gitaly nodes. This metric has two
vectors:
| Metric Name | Vector | What is it? |
|---|---|---|
gitaly_praefect_read_distribution | virtual_storage | The virtual storage name |
gitaly_praefect_read_distribution | storage | The Gitaly storage name |
Pack objects cache
The pack objects cache can be observed through both logs as well as Prometheus metrics.
| Log field name | Description |
|---|---|
pack_objects_cache.hit | Indicates whether the current pack-objects cache was hit (true or false) |
pack_objects_cache.key | Cache key used for the pack-objects cache |
pack_objects_cache.generated_bytes | Size (in bytes) of the new cache being written |
pack_objects_cache.served_bytes | Size (in bytes) of the cache being served |
pack_objects.compression_statistics | Statistics regarding pack-objects generation |
pack_objects.enumerate_objects_ms | Total time (in ms) spent enumerating objects sent by clients |
pack_objects.prepare_pack_ms | Total time (in ms) spent preparing the packfile before sending it back to the client |
pack_objects.write_pack_file_ms | Total time (in ms) spent sending back the packfile to the client. Highly dependent on the client’s internet connection |
pack_objects.written_object_count | Total number of objects Gitaly sends back to the client |
Example log message:
{
"bytes":26186490,
"correlation_id":"01F1MY8JXC3FZN14JBG1H42G9F",
"grpc.meta.deadline_type":"none",
"grpc.method":"PackObjectsHook",
"grpc.request.fullMethod":"/gitaly.HookService/PackObjectsHook",
"grpc.request.glProjectPath":"root/gitlab-workhorse",
"grpc.request.glRepository":"project-2",
"grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git",
"grpc.request.repoStorage":"default",
"grpc.request.topLevelGroup":"@hashed",
"grpc.service":"gitaly.HookService",
"grpc.start_time":"2021-03-25T14:57:52.747Z",
"level":"info",
"msg":"finished unary call with code OK",
"peer.address":"@",
"pid":20961,
"span.kind":"server",
"system":"grpc",
"time":"2021-03-25T14:57:53.543Z",
"pack_objects.compression_statistics": "Total 145991 (delta 68), reused 6 (delta 2), pack-reused 145911",
"pack_objects.enumerate_objects_ms": 170,
"pack_objects.prepare_pack_ms": 7,
"pack_objects.write_pack_file_ms": 786,
"pack_objects.written_object_count": 145991,
"pack_objects_cache.generated_bytes": 49533030,
"pack_objects_cache.hit": "false",
"pack_objects_cache.key": "123456789",
"pack_objects_cache.served_bytes": 49533030,
"peer.address": "127.0.0.1",
"pid": 8813,
}
| Prometheus metric name | Vector | Description |
|---|---|---|
gitaly_pack_objects_served_bytes_total | Size (in bytes) of the cache being served | |
gitaly_pack_objects_cache_lookups_total | result |
hit or miss,indicating whether or not a cache lookup resulted in a cache hit or miss |