Gitaly timeouts and retries

{{< details >}}

• Tier: Free, Premium, Ultimate
• Offering: GitLab Self-Managed

{{< /details >}}

Gitaly provides two types of configurable timeouts:

• Call timeouts, configured by using the GitLab UI.
• Negotiation timeouts, configured by using Gitaly configuration files.

Configure the call timeouts

Configure the following call timeouts to make sure that long-running Gitaly calls don't needlessly take up resources.

Prerequisites:

• Administrator access.

To configure the call timeouts:

1. In the upper-right corner, select Admin.
2. Select Settings > Preferences.
3. Expand the Gitaly timeouts section.
4. Set each timeout as required.

Available call timeouts

Different call timeouts are available for different Gitaly operations.

Configure the negotiation timeouts

{{< history >}}

• Introduced in GitLab 16.5.

{{< /history >}}

You might need to increase the negotiation timeout:

• For particularly large repositories.
• When performing these commands in parallel.

You can configure negotiation timeouts for:

• git-upload-pack(1), which is invoked by a Gitaly node when you execute git fetch.
• git-upload-archive(1), which is invoked by a Gitaly node when you execute git archive --remote.

To configure these timeouts:

{{< tabs >}}

{{< tab title="Linux package (Omnibus)" >}}

Edit /etc/gitlab/gitlab.rb:

gitaly['configuration'] = {
    timeout: {
        upload_pack_negotiation: '10m',      # 10 minutes
        upload_archive_negotiation: '20m',   # 20 minutes
    }
}

{{< /tab >}}

{{< tab title="Self-compiled (source)" >}}

Edit /home/git/gitaly/config.toml:

[timeout]
upload_pack_negotiation = "10m"
upload_archive_negotiation = "20m"

{{< /tab >}}

{{< /tabs >}}

For the values, use the format of ParseDuration in Go.

These timeouts affect only the negotiation phase of remote Git operations, not the entire transfer.

Gitaly client retries

{{< history >}}

• Introduced in GitLab 18.10.

{{< /history >}}

Gitaly can sometimes be briefly unavailable. For example, during GitLab upgrades. Especially with Gitaly on Kubernetes, where a Pod starts and restarts take a couple of seconds.

To prevent GitLab from returning errors to clients when briefly unavailable, configure Gitaly client retries. When Gitaly client retries are configured and Gitaly is unavailable, Gitaly client such as Rails (GitLab application), Workhorse, and GitLab Shell retry request in an exponential backoff fashion.

Two parameters can be configured:

• max_attempts: Maximum number of retry attempts between 1 and 5.
• max_backoff: Maximum amount of time before the client stops retrying. Value must be a duration string, such as 1.4s or 10s.

The backoff multiplier is set to 2 and the initial backoff is derived from the two parameters.

Configuration guidelines

The right configuration depends on your GitLab instance setup and how long Gitaly remains unavailable when such an event occurs:

• On Kubernetes, a Gitaly Pod can take approximately 10 to 12 seconds to start, depending on the Cloud provider. The time includes how long it takes for the volume to be attached and mounted on the Pod.
• For Linux package instances, Gitaly might restart much faster because restarting Gitaly is a process restart.

Also keep in mind is that Gitaly can be configured with a graceful shutdown timeout. When Gitaly is shutting down, new requests are rejected but the gRPC server keeps processing in-flight requests until either:

• They are all served.
• The shutdown timeout lapses.

This graceful shutdown timeout can play a role in how long Gitaly remains unavailable for new requests.

You should configure client retry with a max_backoff that is equal to or greater than sum of the graceful shutdown + the (re)start time.

Configure client retries

The following configuration applies to Rails (GitLab application), Workhorse, and GitLab Shell and the same configuration applies to all clients.

Provided values are examples and should not be treated as guidelines.

{{< tabs >}}

{{< tab title="Linux package (Omnibus)" >}}

Update your gitlab.rb file with these configurations:

gitlab_rails['gitaly_client_max_attempts'] = 5
gitlab_rails['gitaly_client_max_backoff'] = '1.4s'

{{< /tab >}}

{{< tab title="Helm chart (Kubernetes)" >}}

Update your values.yml file with these configurations:

global:
  gitaly:
    client:
      maxAttempts: 5
      maxBackoff: '1.4s'

{{< /tab >}}

{{< /tabs >}}