• Current GitLab.com Architecture
• Future Architecture
• Requirements
  • Universal Requirements
  • Current Platform Requirements
    • Performance
  • Cells Requirements
    • Decomposition
• Overview - Possible Solutions
  • Non-PostgreSQL Offerings
  • Cloud SQL
    • Things to validate
  • k8s Operator
    • Things to validate
  • Refactor Current Automation
• Cells 1.0
• Cells 1.x
• Current Database Platform

Database Platform

Current GitLab.com Architecture

GitLab.com uses 4 PostgreSQL clusters running on virtual machines in GCP managed by Terraform, Chef and Ansible for complex tasks. With Cells, we want to create significant more than 4 PostgreSQL clusters, if we want 50 Cells, we’ll have to have 50+ PostgreSQL clusters.

We currently have 4 different database clusters, each hosting a single production database

The data will be sharded by the organization_id, and an organization_id can live in 1 Cell.

During the Cells rollout most of the data will remain in the current database clusters, so it is important to accommodate for the requirements of this larger monolithic systems as well.

Future Architecture

For the upcoming Cells project the requirements are different. The current estimate is that each Cell will host a relatively small number of users compared to our current monolith, posing the new challenge to host many small database clusters and to create them on short notice

The exact requirements will develop over time, but the following is requested:

• Creation and removal of database clusters via API, which will be part of the provisioning of a cell.
• Database clusters up to the size of our reference architecture, 50,000 users API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS.
• Up to 2,000 Cells and database clusters.

Our current automation involves many manual steps, including the creation of dedicated Chef roles for each cluster. The change process to create new cluster is the largest overhead, requiring multiple Merge Requests and Peer Review as well execution by Change Request. This is incompatible with the newly given requirements.

However, Cells will start iterative with one single Cell, needing one database cluster. The current platform can therefore be used at least for Cells 1.0, but could be kept until all data is stored in the new Cell local database clusters.

It is also possible to migrate the current data to a new database platform, before the full sharing across multiple smaller Cells. If this is desired, the requirements for the current monolith need to be meet by the target platform as well.

Requirements

Requirements will be split into what we currently need for the monolithic data stores for GitLab.com and what we estimate for the individual cells.

Requirements can be put into the following categories. They can be high, soft, low.

• high, is needed to operate GitLab.com.
• medium, is currently needed for operation, but could be replaced
• low, nice to have

Universal Requirements

Current Platform Requirements

Performance

Performance is a key factor. The first task is to define performance outlines and a reliable method to test if they are met.

{- TODO: Define performance requirements and test procedure -}

We observed the following load peaks and can use them as an indication:

• ~100k read-write TPS on the primaries
  • ~80k TPS on the Main primary
  • ~32k TPS on the CI primary
• ~1M read-only TPS distributed among the standbys
  • ~714k TPS on the Main standbys
  • ~150k TPS on the CI standbys

Cells Requirements

{- TODO: Define performance requirements and check with different steak holders. Discussed with @rnienaber, we will start with the largest reference architecture. -}

• Up to around 50.000 users, our largest reference architecture, Up to 50,000 users API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS
• This might increase in the future, or multiple hundred Cells are required to host all current users

Decomposition

We are also evaluating if we can further decompose the Main database with decomposing Secure and Govern related tables to a separate Postgres DB,.

Depending on the actual Cell size and amount of data it makes sense to keep individual clusters for each of the databases, or to use a single cluster per Cell to host multiple databases. To not prematurely limit the maximal size a Cell can have, we should implement the option to uses dedicated clusters for each database.

Currently, Dedicated / GET / Cells tooling does not support decomposition. Adding supporting for Decomposition in GET and Dedicated would be possible as a medium-complexity task. If the given estimates on Cell size are not changed, decomposition with in Cells might not be needed. So for the first iteration only a single database cluster is needed per Cell, only a single one will be supported.

Overview - Possible Solutions

Non-PostgreSQL Offerings

We briefly looked into non-PostgreSQL, claiming certain levels of PostgreSQL compatibility, like AlloyDB. We came to the conclusion that cost and risk are not in any meaningful proportion to the expected benefit. Currently, there is no inherent shortcoming of PostgreSQL itself that justifies further investigation.

Further details below:

• Alternatives To Postgres DB (Google Spanner vs AlloyDB)
• Switching to a proprietary database, instead of PostgreSQL

Cloud SQL

Cloud SQL is Google’s standard PostgreSQL offering. It is a custom fork, claimed to be 100% compatible to the upstream Releases and is simply referenced as PostgreSQL in the Cloud SQL documentation. GitLab currently recognizes Cloud SQL as a supported PostgreSQL implementation.

Most of the information above can be found in the official Cloud SQL documentation. Some estimates were given to us verbally by the GCP team, as seen in the meeting notes Discuss Cloud SQL and AlloyDB with GitLab (internal).

Things to validate

• Could we use the offered logical replication feature (pglogical) for our migration needs? - estimate 2-4 weeks
• Can we access WAL and base_backups, as it appears in the pitr documentation, in contrast to our meeting, where GCP denied it. - estimate < 1 week
• How long does a major upgrade take for our 50k reference architecture? - estimate 4-5 weeks
• Is Query Insights a sufficient replacement for the current observability tooling.
• How long does it take to create a read-replica? How long does it take to create a new cluster from backup? 10GB, 100GB, 1TB - estimate 1 week

k8s Operator

There are currently multiple available operators that seam feasible, but detailed evaluation and benchmarking to ensure our requirements are met is critical. One feature that might be lacking is zero downtime upgrades. Currently, we maintain our own automation for this as well and could adapt it until an operator provides this feature.

Things to validate

• Compare Candidates (4-5 weeks)
  • CYBERTEC PostgreSQL Operator (zalando with commercial support)
  • CloudNativePG

Refactor Current Automation

The current platform to host all PostgreSQL clusters for GitLab.com is based mainly on Terrafrom and Chef, managing GCP virtual machines. Some complex operations are orchestrated via Ansible.

The main problems currently with this architecture is the historical growth and technical debt. It was designed to host a single, or small fixed number of database clusters. Hosting any arbitrary number of clusters is theoretical possible, but requieres manual steps and overhead.

These inefficiencies are incompatible with the estimated requirements of hosting many database clusters, e.g. >100.

Some examples:

• For each database cluster we have to create multiple Chef roles
• For each database cluster we have to manually define a network subnet and write it in a configuration file
• By policy for each database cluster in production, we have an equivalent in staging

These shortcomings and bottlenecks could be removed by improving the automation with Chef or replacing it by a different tool set.

Cells 1.0

Cells 1.0 is the first iteration for Cells, which should become GA during 2024. This first iteration will only host a limited number of test users and projects and therefor does not have the full requirements. It poses an opportunity to learn and iterate on the database platform.

During the development the number of Cells can grow from 1 to up to 10. The estimate is that it will take multiple month until a second Cell will be created, and a second cluster is needed.

Cloud SQL currently does not fulfill the given high priority requirements, and there is no roadmap which fully includes them as well. We will use Cloud SQL temporary for Cells 1.0 despite that to unblock the Cells 1.0 time-line. Therefor we acknowledge and agree to the associated risks and bottlenecks.

• There will be downtime to migrate to the final platform!
• The quality of service will not be equal to GitLab.com.
  • There is no clear SLO for deep level support.
  • No major version upgrades with reasonable downtime.
• We can not grow reliable grow this solution to the target of 50k users, we see problems for 3k users regarding connection pooling already.

We prioritize finding a solution meeting the requirements, to make it available after Cells 1.0. The most promising candidates are the existing k8s PostgreSQL operators, offering PostgreSQL as a Service like experience.

Cells 1.x

As highlighted in Cells 1.0 we start with Cloud SQL as a temporary solution to free resource to build a sufficient platform for the long-term requirements.

The first phase will evaluate which k8s operators are available and which are the most fitting candidates for a full evaluation.

Current Database Platform

The current database platform is limited in future growth, but currently not at the edge of saturation. It is not clear when the database will become a bottleneck, but based on the prior experience from before decomposition a database with 200% size poses operational challenges.

There are currently no predictions how long it takes for our database to double in size, but it is fair to assume this will not happen in the next 24 month, until 2026.

This gives enough time to migrate significant data into cells, or to improve / change the database platform here as well.