ContextQMD
Back to Gitlabhq

Geo with Object storage

doc/administration/geo/replication/object_storage.md

18.11.25.2 KB
Original Source

{{< details >}}

  • Tier: Premium, Ultimate
  • Offering: GitLab Self-Managed

{{< /details >}}

{{< history >}}

  • Verification of files stored in object storage was introduced in GitLab 16.4 with a flag named geo_object_storage_verification. Enabled by default.

{{< /history >}}

Geo can be used in combination with Object Storage (AWS S3, or other compatible object storage).

Secondary sites can use one of the following:

  • The same storage bucket as the primary site.
  • A replicated storage bucket.
  • Local storage, if the primary uses local storage.

The storage method (local or object storage) for files is recorded in the database, and the database is replicated from the primary Geo site to the secondary Geo site.

When accessing an uploaded object, we get its storage method (local or object storage) from the database, so the secondary Geo site must match the storage method of the primary Geo site.

Therefore, if the primary Geo site uses object storage, the secondary Geo site must use it too.

To have:

Read more about using object storage with GitLab.

Object storage verification

Geo verifies files stored in object storage to ensure data integrity between primary and secondary sites.

[!warning] Disabling object storage verification is not recommended. When you disable the geo_object_storage_verification feature flag, GitLab asynchronously deletes all existing verification state records.

When the geo_object_storage_verification feature flag is disabled:

  • Geo verification workers (Geo::VerificationBatchWorker) can still appear in Sidekiq logs, but verification does not take place.
  • During cleanup of verification records, workers may be enqueued to process remaining records.

Enabling GitLab-managed object storage replication

{{< history >}}

{{< /history >}}

[!warning] In case of issues, avoid manually deleting individual files as that can lead to data inconsistencies.

Secondary sites can replicate files stored by the primary site regardless of whether they are stored on the local file system or in object storage.

To enable GitLab replication:

  1. In the upper-right corner, select Admin.
  2. Select Geo > Sites.
  3. Select Edit on the secondary site.
  4. In the Synchronization Settings section, find the Allow this secondary site to replicate content on Object Storage checkbox to enable it.

For LFS, follow the documentation to set up LFS object storage.

For CI job artifacts, there is similar documentation to configure jobs artifact object storage.

For user uploads, there is similar documentation to configure upload object storage.

If you want to migrate the primary site's files to object storage, you can configure the secondary in a few ways:

  • Use the exact same object storage.
  • Use a separate object store but leverage your object storage solution's built-in replication.
  • Use a separate object store and enable the Allow this secondary site to replicate content on Object Storage setting.

If the Allow this secondary site to replicate content on Object Storage setting is disabled, and if you have migrated all your files from local storage to object storage, then many Admin > Geo > Sites progress bars display Nothing to synchronize.

[!warning] To avoid data loss, you should only enable the Allow this secondary site to replicate content on Object Storage setting if you are using separate object stores for the Primary and Secondary sites.

GitLab does not support the case where both:

  • The primary site uses local storage.
  • A secondary site uses object storage.

Inconsistencies after the migration

Data inconsistencies can occur when migrating from local to object storage, which is further described in the object storage troubleshooting section.

Third-party replication services

When using Amazon S3, you can use Cross-Region Replication (CRR) to have automatic replication between the bucket used by the primary site and the bucket used by secondary sites.

If you are using Google Cloud Storage, consider using Multi-Regional Storage. Or you can use the Storage Transfer Service, although this only supports daily synchronization.

For manual synchronization, or scheduled by cron, see: