ContextQMD
Back to Gitlabhq

Configure a new **secondary** site

doc/administration/geo/replication/configuration.md

18.11.214.9 KB
Original Source

{{< details >}}

  • Tier: Premium, Ultimate
  • Offering: GitLab Self-Managed

{{< /details >}}

[!note] This is the final step in setting up a secondary Geo site. Stages of the setup process must be completed in the documented order. If not, complete all prior stages before proceeding.

The basic steps of configuring a secondary site are to:

  1. Replicate required configurations between the primary and the secondary site.
  2. Configure a tracking database on each secondary site.
  3. Start GitLab on each secondary site.

This document focuses on the first item. You are encouraged to first read through all the steps before executing them in your testing/production environment.

Prerequisites for both primary and secondary sites:

[!note] Do not set up any custom authentication for the secondary site. This is handled by the primary site. Any change that requires access to the Admin area needs to be done in the primary site because the secondary site is a read-only replica.

Step 1. Manually replicate secret GitLab values

GitLab stores a number of secret values in the /etc/gitlab/gitlab-secrets.json file which must be the same on all of a site's nodes. Until there is a means of automatically replicating these between sites (see issue #3789), they must be manually replicated to all nodes of the secondary site.

  1. SSH into a Rails node on your primary site, and execute the command below:

    shell
    sudo cat /etc/gitlab/gitlab-secrets.json

    This displays the secrets that need to be replicated, in JSON format.

  2. SSH into each node on your secondary Geo site and login as the root user:

    shell
    sudo -i

  3. Make a backup of any existing secrets:

    shell
    mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F`

  4. Copy /etc/gitlab/gitlab-secrets.json from the Rails node on your primary site to each node on your secondary site, or copy-and-paste the file contents between nodes:

    shell
    sudo editor /etc/gitlab/gitlab-secrets.json

# paste the output of the `cat` command you ran on the primary
# save and exit

  5. Ensure the file permissions are correct:

    shell
    chown root:root /etc/gitlab/gitlab-secrets.json
chmod 0600 /etc/gitlab/gitlab-secrets.json

  6. Reconfigure each Rails, Sidekiq and Gitaly nodes on your secondary site for the change to take effect:

    shell
    gitlab-ctl reconfigure
gitlab-ctl restart

Step 2. Manually replicate the primary site's SSH host keys

GitLab integrates with the system-installed SSH daemon, designating a user (typically named git) through which all access requests are handled.

In a Disaster Recovery situation, GitLab system administrators promote a secondary site to the primary site. DNS records for the primary domain should also be updated to point to the new primary site (previously a secondary site). Doing so avoids the need to update Git remotes and API URLs.

This causes all SSH requests to the newly promoted primary site to fail due to SSH host key mismatch. To prevent this, the primary SSH host keys must be manually replicated to the secondary site.

The SSH host key path depends on the used software:

  • If you use OpenSSH, the path is /etc/ssh.
  • If you use gitlab-sshd, the path is /var/opt/gitlab/gitlab-sshd.

In the following steps, replace <ssh_host_key_path> with the one you're using:

  1. SSH into each Rails node on your secondary site and sign in as the root user:

    shell
    sudo -i

  2. Make a backup of any existing SSH host keys:

    shell
    find <ssh_host_key_path> -iname 'ssh_host_*' -exec cp {} {}.backup.`date +%F` \;

  3. Copy the SSH host keys from the primary site:

    If you can access one of the nodes on your primary site serving SSH traffic (usually, the main GitLab Rails application nodes) using the root user:

    shell
    # Run this from the secondary site, change `<primary_site_fqdn>` for the IP or FQDN of the server
scp root@<primary_node_fqdn>:<ssh_host_key_path>/ssh_host_*_key* <ssh_host_key_path>

    If you only have access through a user with sudo privileges:

    shell
    # Run this from the node on your primary site:
sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz <ssh_host_key_path>/ssh_host_*_key*

# Run this on each node on your secondary site:
scp <user_with_sudo>@<primary_site_fqdn>:geo-host-key.tar.gz .
tar zxvf ~/geo-host-key.tar.gz -C <ssh_host_key_path>

  4. On each Rails node on your secondary site, ensure the file permissions are correct:

    shell
    chown root:root <ssh_host_key_path>/ssh_host_*_key*
chmod 0600 <ssh_host_key_path>/ssh_host_*_key

  5. To verify key fingerprint matches, execute the following command on both primary and secondary nodes on each site:

    shell
    for file in <ssh_host_key_path>/ssh_host_*_key; do ssh-keygen -lf $file; done

    You should get an output similar to this one and they should be identical on both nodes:

    shell
    1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA)
256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA)
256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519)
2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA)

  6. Verify that you have the correct public keys for the existing private keys:

    shell
    # This will print the fingerprint for private keys:
for file in <ssh_host_key_path>/ssh_host_*_key; do ssh-keygen -lf $file; done

# This will print the fingerprint for public keys:
for file in <ssh_host_key_path>/ssh_host_*_key.pub; do ssh-keygen -lf $file; done

    [!note] The output for private keys and public keys command should generate the same fingerprint.

  7. Restart either sshd for OpenSSH or the gitlab-sshd service on each Rails node on your secondary site:

    • For OpenSSH:

      shell
      # Debian or Ubuntu installations
sudo service ssh reload

# CentOS installations
sudo service sshd reload

    • For gitlab-sshd:

      shell
      sudo gitlab-ctl restart gitlab-sshd

  8. Verify SSH is still functional.

    SSH into your GitLab secondary server in a new terminal. If you are unable to connect, verify the permissions are correct according to the previous steps.

Step 3. Add the secondary site

  1. SSH into each Rails and Sidekiq node on your secondary site and login as root:

    shell
    sudo -i

  2. Edit /etc/gitlab/gitlab.rb and add a unique name for your site. You need this in the next steps:

    ruby
    ##
## The unique identifier for the Geo site. See
## https://docs.gitlab.com/administration/geo_sites/#common-settings
##
gitlab_rails['geo_node_name'] = '<site_name_here>'

  3. Reconfigure each Rails and Sidekiq node on your secondary site for the change to take effect:

    shell
    gitlab-ctl reconfigure

  4. Go to the primary node GitLab instance:

    1. In the upper-right corner, select Admin.
    2. In the left sidebar, select Geo > Sites.
    3. Select Add site.
    4. In Name, enter the value for gitlab_rails['geo_node_name'] in /etc/gitlab/gitlab.rb. These values must always match exactly, character for character.
    5. In External URL, enter the value for external_url in /etc/gitlab/gitlab.rb. These values must always match, but it doesn't matter if one ends with a / and the other doesn't.
    6. Optional. In Internal URL (optional), enter an internal URL for the secondary site.
    7. Optional. Select which groups or storage shards should be replicated by the secondary site. Leave blank to replicate all. For more information, see selective synchronization.
    8. Select Save changes to add the secondary site.

  5. SSH into each Rails, and Sidekiq node on your secondary site and restart the services:

    shell
    gitlab-ctl restart

    Check if there are any common issues with your Geo setup by running:

    shell
    gitlab-rake gitlab:geo:check

    If any of the checks fail, check the troubleshooting documentation.

  6. SSH into a Rails or Sidekiq server on your primary site and login as root to verify the secondary site is reachable or there are any common issues with your Geo setup:

    shell
    gitlab-rake gitlab:geo:check

    If any of the checks fail, check the troubleshooting documentation.

After the secondary site is added to the Geo administration page and restarted, the site automatically starts replicating missing data from the primary site in a process known as backfill. Meanwhile, the primary site starts to notify each secondary site of any changes, so that the secondary site can act on those notifications immediately.

Be sure the secondary site is running and accessible. You can sign in to the secondary site with the same credentials as were used with the primary site.

Add primary and secondary URLs as allowed ActionCable origins

This step allows websockets to work seamlessly from primary and secondary sites.

  1. Collect the external URLs of your sites (primary and secondary). You can find them in the Site pages in the Admin area, as mentioned in the section above.

  2. SSH into each Rails and Sidekiq node on your primary site and sign in as root:

    shell
    sudo -i

  3. Edit /etc/gitlab/gitlab.rb to add the URLs collected in step 1 to the action_cable_allowed_origins setting:

    ruby
    gitlab_rails['action_cable_allowed_origins'] = ['https://secondary.example.com', 'https://primary.example.com']

  4. To apply the changes, reconfigure each Rails and Sidekiq node and restart the service:

    shell
    gitlab-ctl reconfigure
gitlab-ctl restart

Step 4. (Optional) Using custom certificates

You can safely skip this step if:

  • Your primary site uses a public CA-issued HTTPS certificate.
  • Your primary site only connects to external services with CA-issued (not self-signed) HTTPS certificates.

Custom or self-signed certificate for inbound connections

If your GitLab Geo primary site uses a custom or self-signed certificate to secure inbound HTTPS connections, this can be either a single-domain or multi-domain certificate.

Install the correct certificate based on your certificate type:

  • Multi-domain certificate that includes both primary and secondary site domains: Install the certificate at /etc/gitlab/ssl on all Rails, Sidekiq, and Gitaly nodes in the secondary site.
  • Single-domain certificate where the certificates are specific to each Geo site domain: Generate a valid certificate for your secondary site's domain and install it at /etc/gitlab/ssl following these instructions on all Rails, Sidekiq, and Gitaly nodes in the secondary site.

Connecting to external services that use custom certificates

A copy of the self-signed certificate for the external service needs to be added to the trust store on all the primary site's nodes that require access to the service.

For the secondary site to be able to access the same external services, these certificates must be added to the secondary site's trust store.

If your primary site is using a custom or self-signed certificate for inbound HTTPS connections, the primary site's certificate needs to be added to the secondary site's trust store:

  1. SSH into each Rails, Sidekiq, and Gitaly node on your secondary site and login as root:

    shell
    sudo -i

  2. Copy the trusted certs from the primary site:

    If you can access one of the nodes on your primary site serving SSH traffic using the root user:

    shell
    scp root@<primary_site_node_fqdn>:/etc/gitlab/trusted-certs/* /etc/gitlab/trusted-certs

    If you only have access through a user with sudo privileges:

    shell
    # Run this from the node on your primary site:
sudo tar --transform 's/.*\///g' -zcvf ~/geo-trusted-certs.tar.gz /etc/gitlab/trusted-certs/*

# Run this on each node on your secondary site:
scp <user_with_sudo>@<primary_site_node_fqdn>:geo-trusted-certs.tar.gz .
tar zxvf ~/geo-trusted-certs.tar.gz -C /etc/gitlab/trusted-certs

  3. Reconfigure each updated Rails, Sidekiq, and Gitaly node in your secondary site:

    shell
    sudo gitlab-ctl reconfigure

Step 5. Verify proper functioning of the secondary site

You can sign in to the secondary site with the same credentials you used with the primary site. After you sign in:

  1. In the upper-right corner, select Admin.
  2. Select Geo > Sites.
  3. Verify that it's correctly identified as a secondary Geo site, and that Geo is enabled.

The initial replication may take some time. The status of the site or the 'backfill' may still in progress. You can monitor the synchronization process on each Geo site from the primary site's Geo Sites dashboard in your browser.

If your installation isn't working properly, check the troubleshooting document.

The two most obvious issues that can become apparent in the dashboard are:

  1. Database replication not working well.
  2. Instance to instance notification not working. In that case, it can be something of the following:
    • You are using a custom certificate or custom CA (see the troubleshooting document).
    • The instance is firewalled (check your firewall rules).

Disabling a secondary site stops the synchronization process.

If repository storages are customized on the primary site for multiple repository shards you must duplicate the same configuration on each secondary site.

Point your users to the Using a Geo Site guide.

Currently, this is what is synced:

  • Git repositories.
  • Wikis.
  • LFS objects.
  • Issues, merge requests, snippets, and comment attachments.
  • Users, groups, and project avatars.

Troubleshooting

See the troubleshooting document.