Geo proxying for secondary sites

{{< details >}}

• Tier: Premium, Ultimate
• Offering: GitLab Self-Managed

{{< /details >}}

{{< history >}}

• HTTP proxying for secondary sites with separate URLs introduced in GitLab 14.5 with a flag named geo_secondary_proxy_separate_urls. Disabled by default.
• Enabled on GitLab.com, GitLab Self-Managed, and GitLab Dedicated in GitLab 15.1.

{{< /history >}}

[!flag] The availability of this feature is controlled by a feature flag. For more information, see the history. The geo_secondary_proxy_separate_urls feature flag is planned to be deprecated and removed in a future release. Support for read-only Geo secondary sites is proposed in issue 366810.

Secondary sites behave as full read-write GitLab instances. They transparently proxy all operations to the primary site, with some notable exceptions.

This behavior enables use-cases including:

• Putting all Geo sites behind a single URL, to deliver a consistent, seamless, and comprehensive experience whichever site the user lands on. Users don't need to juggle multiple GitLab URLs.
• Geographically load-balancing traffic without worrying about write access.

<i class="fa-youtube-play" aria-hidden="true"></i> For an overview, see Geo proxying for secondary sites.

For known issues, see proxying-related items in the Geo documentation.

Set up a unified URL for Geo sites

Secondary sites can transparently serve read-write traffic. Therefore, you can use a single external URL so that requests can hit either the primary Geo site or any secondary Geo sites. This delivers a consistent, seamless, and comprehensive experience whichever site the user lands on. Users don't need to juggle multiple URLs or even be aware of the idea of multiple sites.

You can route traffic to Geo sites with:

• Geo-location aware DNS. To route traffic to the closest Geo site, whether primary or secondary. For an example, follow Configure location-aware DNS.
• Round-robin DNS.
• A load-balancer. It must use sticky sessions to avoid authentication failures and cross-site request errors. DNS routing is inherently sticky so it does not share this caveat.

Configure location-aware DNS

Follow this example to route traffic to the closest Geo site, whether primary or secondary.

Prerequisites

This example creates a gitlab.example.com subdomain that automatically directs requests:

• From Europe to a secondary site.
• From all other locations to the primary site.

For this example, you need:

• A working Geo primary site and secondary site, see the Geo setup instructions.
• A DNS zone managing your domain. Although the following instructions use AWS Route53 and GCP cloud DNS, other services such as Cloudflare can be used as well.

AWS Route53

In this example, you use a Route53 Hosted Zone managing your domain for the Route53 setup.

In a Route53 Hosted Zone, traffic policies can be used to set up a variety of routing configurations. To create a traffic policy:

1. Go to the Route53 dashboard and select Traffic policies.

2. Select Create traffic policy.

3. Fill in the Policy Name field with Single Git Host and select Next.

4. Leave DNS type as A: IP Address in IPv4 format.

5. Select Connect to, then select Geolocation rule.

6. For the first Location:

   1. Leave it as Default.
   2. Select Connect to, then select New endpoint.
   3. Choose Type value and fill it in with <your **primary** IP address>.

7. For the second Location:

   1. Choose Europe.
   2. Select Connect to, then select New endpoint.
   3. Choose Type value and fill it in with <your **secondary** IP address>.

8. Select Create traffic policy.

9. Fill in Policy record DNS name with gitlab.

10. Select Create policy records.

You have successfully set up a single host, like gitlab.example.com, which distributes traffic to your Geo sites by geolocation.

GCP

In this example, you create a GCP Cloud DNS zone managing your domain.

When creating Geo-Based record sets, GCP applies a nearest match for the source region when the source of the traffic doesn't match any policy items exactly. To create a Geo-Based record set:

1. Select Network Services > Cloud DNS.
2. Select the Zone configured for your domain.
3. Select Add Record Set.
4. Enter the DNS Name for your Location-aware public URL, for example, gitlab.example.com.
5. Select the Routing Policy: Geo-Based.
6. Select Add Managed RRData.
   1. Select Source Region: us-central1.
   2. Enter your <**primary** IP address>.
   3. Select Done.
7. Select Add Managed RRData.
   1. Select Source Region: europe-west1.
   2. Enter your <**secondary** IP address>.
   3. Select Done.
8. Select Create.

You have successfully set up a single host, like gitlab.example.com, which distributes traffic to your Geo sites using a location-aware URL.

Configure each site to use the same external URL

After you have set up routing from a single URL to all of your Geo sites, follow the following steps if your sites use different URLs:

1. On each GitLab site, SSH into each node running Rails (Puma, Sidekiq, Log-Cursor) and set the external_url to that of the single URL:

   shellCopy

   sudo -e /etc/gitlab/gitlab.rb
   

2. Reconfigure the updated nodes for the change to take effect:

   shellCopy

   sudo gitlab-ctl reconfigure
   

3. To match the new external URL set on the secondary Geo sites, the primary database needs to reflect this change.

   In the Geo administration page of the primary site, edit each Geo secondary that is using the secondary proxying and set the URL field to the single URL. Make sure the primary site is also using this URL.

   To allow the sites to talk to each other, make sure the Internal URL field is unique for each site.

In Kubernetes, you can use the same domain under global.hosts.domain as for the primary site.

Set up a separate URL for a secondary Geo site

You can use different external URLs per site. You can use this to offer a specific site to a specific set of users. Alternatively, you can give users control over which site they use, though they must understand the implications of their choice.

[!note] GitLab does not support multiple external URLs, see issue 21319. An inherent problem is there are many cases where a site needs to produce an absolute URL outside of the context of an HTTP request, such as when sending emails that were not triggered by a request.

Configure a secondary Geo site to a different external URL than the primary site

If your secondary site uses the same external URL as the primary site, but you want to change it to use a different URL:

1. On the secondary site, SSH into each node running Rails (Puma, Sidekiq, Log-Cursor) and set the external_url to the desired URL for the secondary site:

   shellCopy

   sudo -e /etc/gitlab/gitlab.rb
   

2. Reconfigure the updated nodes for the change to take effect:

   shellCopy

   sudo gitlab-ctl reconfigure
   

3. To match the new external URL set on the secondary Geo site, the primary database needs to reflect this change.

   In the Geo administration page of the primary site, edit the target secondary site and set the URL field to the desired URL.

   To allow the sites to talk to each other, make sure the Internal URL field is unique for each site. If the desired URL is unique to this site, then you can clear the Internal URL field. On save, it defaults to the external URL.

Behavior of secondary sites when the primary Geo site is down

Considering that web traffic is proxied to the primary, the behavior of the secondary sites differs when the primary site is inaccessible:

• UI and API traffic return the same errors as the primary (or fail if the primary is not accessible at all) because they are proxied.
• For repositories that are fully up-to-date on the specific secondary site being accessed, Git read operations still work as expected, including authentication through HTTP(s) or SSH. However, Git reads performed by GitLab Runners will fail.
• Git operations for repositories that are not replicated to the secondary site return the same errors as the primary site because they are proxied.
• All Git write operations return the same errors as the primary site because they are proxied.

Features accelerated by secondary Geo sites

Most HTTP traffic sent to a secondary Geo site is proxied to the primary Geo site. With this architecture, secondary Geo sites are able to support write requests, and avoid read-after-write problems. Certain read requests are handled locally by secondary sites for improved latency and bandwidth nearby.

The following table details the components tested through the Geo secondary site Workhorse proxy. It does not cover all data types.

In this context, accelerated reads refer to read requests served from the secondary site, provided that the data is up to date for the component on the secondary site. If the data on the secondary site is determined to be out of date, the request is forwarded to the primary site. Read requests for components not listed in the table below are always automatically forwarded to the primary site.

To request acceleration of a feature, check if an issue already exists in epic 8239 and upvote or comment on it to indicate your interest or ask your GitLab representative to do so on your behalf. If an applicable issue doesn't exist, open one and mention it in the epic.

Disable secondary site HTTP proxying

Secondary site HTTP proxying is enabled by default on a secondary site when it uses a unified URL, meaning, it is configured with the same external_url as the primary site. Disabling proxying in this case tends not to be helpful due to completely different behavior being served at the same URL, depending on routing. When HTTP proxying is disabled on a secondary Geo site, the site operates in read-only mode, with several important limitations you should be aware of.

What happens if you disable secondary proxying

Disabling the proxying feature flag has the following general effects.

HTTP and Git requests

• The secondary site does not proxy HTTP requests to the primary site. Instead, it attempts to serve them itself, or fail.
• Git requests generally succeed. Git pushes are redirected or proxied to the primary site.
• Other than Git requests, any HTTP request which may write data fails. Read requests generally succeed.

You should use the feature flag over using the GEO_SECONDARY_PROXY environment variable.

HTTP proxying is enabled by default in GitLab 15.1 on a secondary site even without a unified URL.

Terms of service acceptance

When proxying is disabled, users who access only the secondary site cannot properly accept terms of service or other legal agreements. This creates the following issues:

• No record of acceptance: If an employee only logs into the secondary site, their acceptance of terms and conditions is not recorded in the primary database because write operations (including terms acceptance) are not proxied when secondary proxying is disabled, even though they may be presented with the terms message.
• Legal compliance concerns: Organizations may lack proper legal coverage if employees use GitLab services through a secondary-only access pattern, since there's no verifiable record of their agreement to the terms and conditions.

As a workaround, you must access the primary site at least once to properly accept terms and conditions. After accepted on the primary, this information is replicated to secondary sites through normal Geo synchronization.

[!note] This limitation affects organizations that require documented acceptance of terms and conditions for compliance or legal purposes. Ensure users have access to the primary site for the initial terms acceptance.

Disable proxy on all secondary sites

If you need to disable proxying on all secondary sites, it is easiest to disable the feature flag:

{{< tabs >}}

{{< tab title="Linux package (Omnibus)" >}}

1. SSH into a node which is running Puma or Sidekiq on your primary Geo site and run:

   shellCopy

   sudo gitlab-rails runner "Feature.disable(:geo_secondary_proxy_separate_urls)"
   

2. Restart Puma on all nodes which are running it on your secondary Geo site:

   shellCopy

   sudo gitlab-ctl restart puma
   

{{< /tab >}}

{{< tab title="Helm chart (Kubernetes)" >}}

1. On your primary Geo site, run this command in the Toolbox pod:

   shellCopy

   kubectl exec -it <toolbox-pod-name> -- gitlab-rails runner "Feature.disable(:geo_secondary_proxy_separate_urls)"
   

2. Restart the Webservice pods on your secondary Geo site:

   shellCopy

   kubectl rollout restart deployment -l app=webservice
   

{{< /tab >}}

{{< /tabs >}}

To revert the changes so secondary site proxying is enabled again:

{{< tabs >}}

{{< tab title="Linux package (Omnibus)" >}}

1. SSH into a node which is running Puma or Sidekiq on your primary Geo site and run:

   shellCopy

   sudo gitlab-rails runner "Feature.enable(:geo_secondary_proxy_separate_urls)"
   

2. Restart Puma on all nodes which are running it on your secondary Geo site:

   shellCopy

   sudo gitlab-ctl restart puma
   

{{< /tab >}}

{{< tab title="Helm chart (Kubernetes)" >}}

1. On your primary Geo site, run this command in the Toolbox pod:

   shellCopy

   kubectl exec -it <toolbox-pod-name> -- gitlab-rails runner "Feature.enable(:geo_secondary_proxy_separate_urls)"
   

2. Restart the Webservice pods on your secondary Geo site:

   shellCopy

   kubectl rollout restart deployment -l app=webservice
   

{{< /tab >}}

{{< /tabs >}}

Disable secondary site HTTP proxying per site

If there are multiple secondary sites, you can disable HTTP proxying on each secondary site separately, by following these steps:

{{< tabs >}}

{{< tab title="Linux package (Omnibus)" >}}

1. SSH into each application node (serving user traffic directly) on your secondary Geo site and add the following environment variable:

   shellCopy

   sudo -e /etc/gitlab/gitlab.rb
   

   rubyCopy

   gitlab_workhorse['env'] = {
     "GEO_SECONDARY_PROXY" => "0"
   }
   

2. Reconfigure the updated nodes for the change to take effect:

   shellCopy

   sudo gitlab-ctl reconfigure
   

{{< /tab >}}

{{< tab title="Helm chart (Kubernetes)" >}}

You can use --set gitlab.webservice.extraEnv.GEO_SECONDARY_PROXY="0", or specify the following in your values file:

gitlab:
  webservice:
    extraEnv:
      GEO_SECONDARY_PROXY: "0"

{{< /tab >}}

{{< /tabs >}}

Disable secondary site Git proxying

It is not possible to disable forwarding of:

• Git push over SSH
• Git pull over SSH when the Git repository is out-of-date on the secondary site
• Git push over HTTP
• Git pull over HTTP when the Git repository is out-of-date on the secondary site