Database load balancing

With database load balancing, read-only queries can be distributed across multiple PostgreSQL nodes to increase performance.

This documentation provides a technical overview on how database load balancing is implemented in GitLab Rails and Sidekiq.

Nomenclature

1. Host: Each database host. It could be a primary or a replica.
2. Primary: Primary PostgreSQL host that is used for write-only and read-and-write operations.
3. Replica: Secondary PostgreSQL hosts that are used for read-only operations.
4. Workload: a Rails request or a Sidekiq job that requires database connections.

Components

A few Ruby classes are involved in the load balancing process. All of them are in the namespace Gitlab::Database::LoadBalancing:

1. Host
2. LoadBalancer
3. ConnectionProxy
4. Session

Each workload begins with a new instance of Gitlab::Database::LoadBalancing::Session. The Session keeps track of the database operations that have been performed. It then determines if the workload requires a connection to either the primary host or a replica host.

When the workload requires a database connection through ActiveRecord, ConnectionProxy first redirects the connection request to LoadBalancer. ConnectionProxy requests either a read or read_write connection from the LoadBalancer depending on a few criteria:

1. Whether the query is a read-only or it requires write.
2. Whether the Session has recorded a write operation previously.
3. Whether any special blocks have been used to prefer primary or replica, such as:
   • use_primary
   • ignore_writes
   • use_replicas_for_read_queries
   • fallback_to_replicas_for_ambiguous_queries

LoadBalancer then yields the requested connection from the respective database connection pool. It yields either:

• A read_write connection from the primary's connection pool.
• A read connection from the replicas' connection pools.

When responding to a request for a read connection, LoadBalancer would first attempt to load balance the connection across the replica hosts. It looks for the next online replica host and yields a connection from the host's connection pool. A replica host is considered online if it is up-to-date with the primary, based on either the replication lag size or time. The thresholds for these requirements are configurable.

Deployment Strategy

When rolling out changes via feature flag, consider deploying exclusively to Sidekiq pods initially to minimize risk.

Why Sidekiq-first deployment:

• Keeps the API pods stable ensuring ChatOps remains available to disable feature flags in worst case scenario.
• Background jobs can retry automatically without any intervention.

Implementation example:

if feature_flag_enabled? && Gitlab::Runtime.sidekiq?
   new_changes
else
   existing_changes
end