Object storage

{{< details >}}

• Tier: Free, Premium, Ultimate
• Offering: GitLab Self-Managed

{{< /details >}}

GitLab supports using an object storage service for holding numerous types of data. It's recommended over NFS and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable.

To configure the object storage, you have two options:

• Recommended. Configure a single storage connection for all object types: A single credential is shared by all supported object types. This is called the consolidated form.

• Configure each object type to define its own storage connection: Every object defines its own object storage connection and configuration. This is called the storage-specific form.

  If you already use the storage-specific form, see how to transition to the consolidated form.

If you store data locally, see how to migrate to object storage.

Object storage provider support

GitLab uses the Fog library for object storage and supports the following three connection types. Other Fog providers are not supported.

If your object storage service is compatible with one of these connection types, configure it using the corresponding connection settings below. The choice of provider is yours.

Providers with active test coverage

GitLab actively tests the following providers:

• Amazon S3 - AWS connection type. Object Lock is not supported. For more information, see issue 335775.
• Google Cloud Storage - Google connection type.
• Azure Blob Storage - AzureRM connection type.

Community-documented providers

The following providers have been documented by the community. GitLab does not test these providers. Configuration examples are provided as a convenience. If you use one of these providers and encounter issues, GitLab Support might be unable to help.

• Digital Ocean Spaces. S3-compatible, see provider-specific configuration examples.
• Oracle Cloud Infrastructure. S3-compatible, see provider-specific configuration examples.
• OpenStack Swift (S3-compatible mode).
• Storj Gateway. S3-compatible, see provider-specific configuration examples.
• Ceph RGW. S3-compatible, see provider-specific configuration examples
• Hitachi Vantara HCP. S3-compatible, see provider-specific configuration examples.
• On-premises hardware and appliances that expose an S3-compatible API.

Configure a single storage connection for all object types (consolidated form)

Most types of objects, such as CI artifacts, LFS files, and upload attachments can be saved in object storage by specifying a single credential for object storage with multiple buckets.

[!note] For GitLab Helm Charts, see how to configure the consolidated form.

Configuring the object storage using the consolidated form has a number of advantages:

• It can simplify your GitLab configuration because the connection details are shared across object types.
• It enables the use of encrypted S3 buckets.
• It uploads files to S3 with proper Content-MD5 headers.

When the consolidated form is used, direct upload is enabled automatically. Thus, only the following providers can be used:

• S3-compatible providers
• Google Cloud Storage
• Azure Blob storage

The consolidated form configuration can't be used for backups or Mattermost. Backups can be configured with server side encryption separately. See the table for a complete list of supported object storage types.

Enabling the consolidated form enables object storage for all object types. If not all buckets are specified, you may see an error like:

Object storage for <object type> must have a bucket specified

If you want to use local storage for specific object types, you can disable object storage for specific features.

Configure the common parameters

In the consolidated form, the object_store section defines a common set of parameters.

For an example, see how to use the consolidated form and Amazon S3.

Configure the parameters of each object

Each object type must at least define the bucket name where it will be stored.

The following table lists the valid objects that can be used:

Within each object type, three parameters can be defined:

For an example, see how to use the consolidated form and Amazon S3.

Disable object storage for specific features

As seen previously, object storage can be disabled for specific types by setting the enabled flag to false. For example, to disable object storage for CI artifacts:

gitlab_rails['object_store']['objects']['artifacts']['enabled'] = false

A bucket is not needed if the feature is disabled entirely. For example, no bucket is needed if CI artifacts are disabled with this setting:

gitlab_rails['artifacts_enabled'] = false

Configure each object type to define its own storage connection (storage-specific form)

With the storage-specific form, every object defines its own object storage connection and configuration. You should use the consolidated form instead, except for the storage types not supported by the consolidated form. When working with the GitLab Helm charts, refer to how the charts handle consolidated form for object storage.

The use of encrypted S3 buckets with non-consolidated form is not supported. You may get ETag mismatch errors if you use it.

[!note] For the storage-specific form, direct upload may become the default because it does not require a shared folder.

For storage types not supported by the consolidated form, refer to the following guides:

Configure the connection settings

Both consolidated and storage-specific form must configure a connection. The following sections describe parameters that can be used in the connection setting.

S3-compatible providers

These settings apply to Amazon S3 and any S3-compatible service using the AWS connection type. When not using AWS directly, set endpoint to your provider's URL.

S3-compatible services vary in how closely they implement the AWS S3 API. GitLab uses specific S3 behaviours, including pre-signed URLs, multipart uploads, and optionally chunked signature streaming, that not every S3-compatible implementation supports identically. If a provider works in other tools but not in GitLab, the settings that most likely need adjustment are:

• aws_signature_version.
• enable_signature_v4_streaming.

The connection settings match those provided by fog-aws:

S3 compatibility and known failure modes

Claiming S3 compatibility does not guarantee a provider works correctly with GitLab. If you encounter errors with an S3-compatible provider, try the following adjustments before raising a support request:

• Signature streaming: Some providers reject chunked transfer encoding used by AWS Signature Version 4 streaming. Set enable_signature_v4_streaming: false.
• Signature version: Some providers do not fully support Signature Version 4. Set aws_signature_version: 2.
• Path-style URLs: Some providers require path-style bucket addressing. Set path_style: true.
• ETag validation: Some providers return ETags that do not match the MD5 of the uploaded object, which GitLab validates. See ETag mismatch.

GitLab Support can help troubleshoot configuration, but cannot guarantee resolution for issues specific to providers not in the tested set.

Use Amazon instance profiles

Instead of supplying AWS access and secret keys in object storage configuration, you can configure GitLab to use Amazon Identity Access and Management (IAM) roles to set up an Amazon instance profile. When this is used, GitLab fetches temporary credentials each time an S3 bucket is accessed, so no hard-coded values are needed in the configuration.

Prerequisites:

• GitLab must be able to connect to the instance metadata endpoint.
• If GitLab is configured to use an internet proxy, the endpoint IP address must be added to the no_proxy list.
• For IMDS v2 access, ensure the hop limit is sufficient. If GitLab is running in a container, you may need to raise the limit from 1 to 2.

To set up an instance profile:

1. Create an IAM role with the necessary permissions. The following example is a role for an S3 bucket named test-bucket:

   jsonCopy

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:PutObject",
                   "s3:GetObject",
                   "s3:DeleteObject"
               ],
               "Resource": "arn:aws:s3:::test-bucket/*"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket"
               ],
               "Resource": "arn:aws:s3:::test-bucket"
           }
       ]
   }
   

2. Attach this role to the EC2 instance hosting your GitLab instance.

3. Set the use_iam_profile GitLab configuration option to true.

Encrypted S3 buckets

When configured either with an instance profile or with the consolidated form, GitLab Workhorse properly uploads files to S3 buckets that have SSE-S3 or SSE-KMS encryption enabled by default. AWS KMS keys and SSE-C encryption are not supported because this requires sending the encryption keys in every request.

Server-side encryption headers

Setting a default encryption on an S3 bucket is the easiest way to enable encryption, but you may want to set a bucket policy to ensure only encrypted objects are uploaded. To do this, you must configure GitLab to send the proper encryption headers in the storage_options configuration section:

As with the case for default encryption, these options only work when the Workhorse S3 client is enabled. One of the following two conditions must be fulfilled:

• use_iam_profile is true in the connection settings.
• Consolidated form is in use.

ETag mismatch errors occur if server side encryption headers are used without enabling the Workhorse S3 client.

Google Cloud Storage (GCS)

{{< history >}}

• universe_domain setting introduced in GitLab 18.9.

{{< /history >}}

Here are the valid connection parameters for GCS:

GitLab reads the value of google_json_key_location, then google_json_key_string, and finally, google_application_default. It uses the first of these settings that has a value.

The service account must have permission to access the bucket. For more information, see the Cloud Storage authentication documentation.

Google Cloud Application Default Credentials

Google Cloud Application Default Credentials (ADC) are typically used with GitLab to use the default service account or Workload Identity Federation. Set google_application_default to true and omit google_json_key_location and google_json_key_string.

If you use ADC, be sure that:

• The service account that you use has the iam.serviceAccounts.signBlob permission. Typically this is done by granting the Service Account Token Creator role to the service account.

• If you are using Google Compute virtual machines, ensure they have the correct access scopes to access Google Cloud APIs. If the machines do not have the right scope, the error logs may show:

  shellCopy

  Google::Apis::ClientError (insufficientPermissions: Request had insufficient authentication scopes.)
  

[!note] To use bucket encryption with customer-managed encryption keys, use the consolidated form.

{{< tabs >}}

{{< tab title="Linux package (Omnibus)" >}}

1. Edit /etc/gitlab/gitlab.rb and add the following lines, substituting the values you want:

   rubyCopy

   gitlab_rails['object_store']['connection'] = {
    'provider' => 'Google',
    'google_project' => '<GOOGLE PROJECT>',
    'google_json_key_location' => '<FILENAME>'
   }
   

   To use ADC, use google_application_default instead:

   rubyCopy

   gitlab_rails['object_store']['connection'] = {
    'provider' => 'Google',
    'google_project' => '<GOOGLE PROJECT>',
    'google_application_default' => true
   }
   

   To use a non-default universe domain (for example, Google Cloud Dedicated):

   rubyCopy

   gitlab_rails['object_store']['connection'] = {
    'provider' => 'Google',
    'google_project' => '<GOOGLE PROJECT>',
    'google_application_default' => true,
    'universe_domain' => '<UNIVERSE DOMAIN>'
   }
   

2. Save the file and reconfigure GitLab:

   shellCopy

   sudo gitlab-ctl reconfigure
   

{{< /tab >}}

{{< tab title="Helm chart (Kubernetes)" >}}

1. Put the following content in a file named object_storage.yaml to be used as a Kubernetes Secret:

   yamlCopy

   provider: Google
   google_project: <GOOGLE PROJECT>
   google_json_key_location: '<FILENAME>'
   

   To use ADC, use google_application_default instead:

   yamlCopy

   provider: Google
   google_project: <GOOGLE PROJECT>
   google_application_default: true
   

   To use a non-default universe domain (for example, Google Cloud Dedicated):

   yamlCopy

   provider: Google
   google_project: <GOOGLE PROJECT>
   google_application_default: true
   universe_domain: <UNIVERSE DOMAIN>
   

2. Create the Kubernetes Secret:

   shellCopy

   kubectl create secret generic -n <namespace> gitlab-object-storage --from-file=connection=object_storage.yaml
   

3. Export the Helm values:

   shellCopy

   helm get values gitlab > gitlab_values.yaml
   

4. Edit gitlab_values.yaml:

   yamlCopy

   global:
     appConfig:
        artifacts:
          bucket: gitlab-artifacts
        ciSecureFiles:
          bucket: gitlab-ci-secure-files
          enabled: true
        dependencyProxy:
          bucket: gitlab-dependency-proxy
          enabled: true
        externalDiffs:
          bucket: gitlab-mr-diffs
          enabled: true
        lfs:
          bucket: gitlab-lfs
        object_store:
          connection:
            secret: gitlab-object-storage
          enabled: true
          proxy_download: false
        packages:
          bucket: gitlab-packages
        terraformState:
          bucket: gitlab-terraform-state
          enabled: true
        uploads:
          bucket: gitlab-uploads
   

5. Save the file and apply the new values:

   shellCopy

   helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
   

{{< /tab >}}

{{< tab title="Docker" >}}

1. Edit docker-compose.yml:

   yamlCopy

   version: "3.6"
   services:
     gitlab:
       environment:
         GITLAB_OMNIBUS_CONFIG: |
           # Consolidated object storage configuration
           gitlab_rails['object_store']['enabled'] = true
           gitlab_rails['object_store']['proxy_download'] = false
           gitlab_rails['object_store']['connection'] = {
             'provider' => 'Google',
             'google_project' => '<GOOGLE PROJECT>',
             'google_json_key_location' => '<FILENAME>'
           }
           gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
           gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs'
           gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'
           gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads'
           gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
           gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
           gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
           gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
           gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
   

   To use ADC, use google_application_default instead:

   rubyCopy

   gitlab_rails['object_store']['connection'] = {
     'provider' => 'Google',
     'google_project' => '<GOOGLE PROJECT>',
     'google_application_default' => true
   }
   

   To use a non-default universe domain (for example, Google Cloud Dedicated):

   rubyCopy

   gitlab_rails['object_store']['connection'] = {
     'provider' => 'Google',
     'google_project' => '<GOOGLE PROJECT>',
     'google_application_default' => true,
     'universe_domain' => '<UNIVERSE DOMAIN>'
   }
   

2. Save the file and restart GitLab:

   shellCopy

   docker compose up -d
   

{{< /tab >}}

{{< /tabs >}}

Azure Blob storage

Although Azure uses the word container to denote a collection of blobs, GitLab standardizes on the term bucket. Be sure to configure Azure container names in the bucket settings.

Azure Blob storage can only be used with the consolidated form because a single set of credentials are used to access multiple containers. The storage-specific form is not supported. For more details, see how to transition to consolidated form.

The following are the valid connection parameters for Azure. For more information, see the Azure Blob Storage documentation.

{{< tabs >}}

{{< tab title="Linux package (Omnibus)" >}}

1. Edit /etc/gitlab/gitlab.rb and add the following lines, substituting the values you want:

   rubyCopy

   gitlab_rails['object_store']['connection'] = {
     'provider' => 'AzureRM',
     'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>',
     'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>',
     'azure_storage_domain' => '<AZURE STORAGE DOMAIN>'
   }
   gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
   gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs'
   gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'
   gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads'
   gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
   gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
   gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
   gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
   gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
   

   If you are using a workload identity, omit azure_storage_access_key:

   rubyCopy

   gitlab_rails['object_store']['connection'] = {
     'provider' => 'AzureRM',
     'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>',
     'azure_storage_domain' => '<AZURE STORAGE DOMAIN>'
   }
   

2. Save the file and reconfigure GitLab:

   shellCopy

   sudo gitlab-ctl reconfigure
   

{{< /tab >}}

{{< tab title="Helm chart (Kubernetes)" >}}

1. Put the following content in a file named object_storage.yaml to be used as a Kubernetes Secret:

   yamlCopy

   provider: AzureRM
   azure_storage_account_name: <YOUR_AZURE_STORAGE_ACCOUNT_NAME>
   azure_storage_access_key: <YOUR_AZURE_STORAGE_ACCOUNT_KEY>
   azure_storage_domain: blob.core.windows.net
   

   If you are using a workload or managed identity, omit azure_storage_access_key:

   yamlCopy

   provider: AzureRM
   azure_storage_account_name: <YOUR_AZURE_STORAGE_ACCOUNT_NAME>
   azure_storage_domain: blob.core.windows.net
   

2. Create the Kubernetes Secret:

   shellCopy

   kubectl create secret generic -n <namespace> gitlab-object-storage --from-file=connection=object_storage.yaml
   

3. Export the Helm values:

   shellCopy

   helm get values gitlab > gitlab_values.yaml
   

4. Edit gitlab_values.yaml:

   yamlCopy

   global:
     appConfig:
        artifacts:
          bucket: gitlab-artifacts
        ciSecureFiles:
          bucket: gitlab-ci-secure-files
          enabled: true
        dependencyProxy:
          bucket: gitlab-dependency-proxy
          enabled: true
        externalDiffs:
          bucket: gitlab-mr-diffs
          enabled: true
        lfs:
          bucket: gitlab-lfs
        object_store:
          connection:
            secret: gitlab-object-storage
          enabled: true
          proxy_download: false
        packages:
          bucket: gitlab-packages
        terraformState:
          bucket: gitlab-terraform-state
          enabled: true
        uploads:
          bucket: gitlab-uploads
   

5. Save the file and apply the new values:

   shellCopy

   helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
   

{{< /tab >}}

{{< tab title="Docker" >}}

1. Edit docker-compose.yml:

   yamlCopy

   version: "3.6"
   services:
     gitlab:
       environment:
         GITLAB_OMNIBUS_CONFIG: |
           # Consolidated object storage configuration
           gitlab_rails['object_store']['enabled'] = true
           gitlab_rails['object_store']['proxy_download'] = false
           gitlab_rails['object_store']['connection'] = {
             'provider' => 'AzureRM',
             'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>',
             'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>',
             'azure_storage_domain' => '<AZURE STORAGE DOMAIN>'
           }
           gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
           gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs'
           gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'
           gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads'
           gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
           gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
           gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
           gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
           gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
   

   If you are using a managed identity, omit azure_storage_access_key.

   rubyCopy

   gitlab_rails['object_store']['connection'] = {
     'provider' => 'AzureRM',
     'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>',
     'azure_storage_domain' => '<AZURE STORAGE DOMAIN>'
   }
   

2. Save the file and restart GitLab:

   shellCopy

   docker compose up -d
   

{{< /tab >}}

{{< tab title="Self-compiled (source)" >}}

For self-compiled installations, Workhorse also needs to be configured with Azure credentials. This isn't needed in Linux package installations because the Workhorse settings are populated from the previous settings.

1. Edit /home/git/gitlab/config/gitlab.yml and add or amend the following lines:

   yamlCopy

   production: &base
     object_store:
       enabled: true
       proxy_download: false
       connection:
         provider: AzureRM
         azure_storage_account_name: '<AZURE STORAGE ACCOUNT NAME>'
         azure_storage_access_key: '<AZURE STORAGE ACCESS KEY>'
       objects:
         artifacts:
           bucket: gitlab-artifacts
         external_diffs:
           bucket: gitlab-mr-diffs
         lfs:
           bucket: gitlab-lfs
         uploads:
           bucket: gitlab-uploads
         packages:
           bucket: gitlab-packages
         dependency_proxy:
           bucket: gitlab-dependency-proxy
         terraform_state:
           bucket: gitlab-terraform-state
         ci_secure_files:
           bucket: gitlab-ci-secure-files
         pages:
           bucket: gitlab-pages
   

2. Edit /home/git/gitlab-workhorse/config.toml and add or amend the following lines:

   tomlCopy

   [object_storage]
     provider = "AzureRM"
   
   [object_storage.azurerm]
     azure_storage_account_name = "<AZURE STORAGE ACCOUNT NAME>"
     azure_storage_access_key = "<AZURE STORAGE ACCESS KEY>"
   

   If you are using a custom Azure storage domain, azure_storage_domain does not have to be set in the Workhorse configuration. This information is exchanged in an API call between GitLab Rails and Workhorse.

3. Save the file and restart GitLab:

   shellCopy

   # For systems running systemd
   sudo systemctl restart gitlab.target
   
   # For systems running SysV init
   sudo service gitlab restart
   

{{< /tab >}}

{{< /tabs >}}

Azure workload and managed identities

{{< history >}}

• Introduced in GitLab 17.9

{{< /history >}}

To use Azure workload identities or managed identities, omit azure_storage_access_key from the configuration. When azure_storage_access_key is blank, GitLab attempts to:

1. Obtain temporary credentials with a workload identity. AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_FEDERATED_TOKEN_FILE should be in the environment.
2. If a workload identity is not available, request credentials from the Azure Instance Metadata Service.
3. Get a User Delegation Key.
4. Generate a SAS token with that key to access a Storage Account blob.

Ensure that the identity has the Storage Blob Data Contributor role assigned to it.

Provider-specific configuration examples

The following examples show configuration for specific S3-compatible providers that require non-default settings. For any S3-compatible provider not listed here, use the base S3-compatible configuration with the appropriate endpoint for your provider.

Oracle Cloud Infrastructure

Oracle Cloud Infrastructure S3 requires the following settings:

If enable_signature_v4_streaming is set to true, you may see the following error in production.log:

STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported

Storj Gateway (SJ)

[!note] The Storj Gateway does not support multi-threaded copying (see UploadPartCopy in the table). While an implementation is planned, you must disable multi-threaded copying until completion.

The Storj Network provides an S3-compatible API gateway. Use the following configuration example:

gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  'endpoint' => 'https://gateway.storjshare.io',
  'path_style' => true,
  'region' => 'eu1',
  'aws_access_key_id' => 'ACCESS_KEY',
  'aws_secret_access_key' => 'SECRET_KEY',
  'aws_signature_version' => 2,
  'enable_signature_v4_streaming' => false
}

The signature version must be 2. Using v4 results in an HTTP 411 Length Required error. For more information, see issue #4419.

Hitachi Vantara HCP

[!note] Connections to HCP may return an error stating SignatureDoesNotMatch - The request signature we calculated does not match the signature you provided. Check your HCP Secret Access key and signing method. In these cases, set the endpoint to the URL of the tenant instead of the namespace, and ensure bucket paths are configured as <namespace_name>/<bucket_name>.

HCP provides an S3-compatible API. Use the following configuration example:

gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  'endpoint' => 'https://<tenant_endpoint>',
  'path_style' => true,
  'region' => 'eu1',
  'aws_access_key_id' => 'ACCESS_KEY',
  'aws_secret_access_key' => 'SECRET_KEY',
  'aws_signature_version' => 4,
  'enable_signature_v4_streaming' => false
}

# Example of <namespace_name/bucket_name> formatting
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = '<namespace_name>/<bucket_name>'

Ceph RGW

Ceph RGW is an S3-compatible API for Ceph. Use the following configuration example:

gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  'endpoint' => 'https://rgw-ceph.example.com',
  'region' => 'us-west-1',
  'aws_access_key_id' => 'ACCESS_KEY',
  'aws_secret_access_key' => 'SECRET_KEY',
  'path_style': true
}

To enable server-side encryption with Ceph RGW, you must connect using HTTPS. Ceph rejects encryption requests over non-secure connections.

Full example using the consolidated form and Amazon S3

The following example uses AWS S3 to enable object storage for all supported services:

{{< tabs >}}

{{< tab title="Linux package (Omnibus)" >}}

1. Edit /etc/gitlab/gitlab.rb and add the following lines, substituting the values you want:

   rubyCopy

   # Consolidated object storage configuration
   gitlab_rails['object_store']['enabled'] = true
   gitlab_rails['object_store']['proxy_download'] = false
   gitlab_rails['object_store']['connection'] = {
     'provider' => 'AWS',
     'region' => 'eu-central-1',
     'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>',
     'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>'
   }
   # OPTIONAL: The following lines are only needed if server side encryption is required
   gitlab_rails['object_store']['storage_options'] = {
     'server_side_encryption' => '<AES256 or aws:kms>',
     'server_side_encryption_kms_key_id' => '<arn:aws:kms:xxx>'
   }
   gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
   gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs'
   gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'
   gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads'
   gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
   gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
   gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
   gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
   gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
   

   If you're using AWS IAM profiles, omit the AWS access key and secret access key/value pairs. For example:

   rubyCopy

   gitlab_rails['object_store']['connection'] = {
     'provider' => 'AWS',
     'region' => 'eu-central-1',
     'use_iam_profile' => true
   }
   

2. Save the file and reconfigure GitLab:

   shellCopy

   sudo gitlab-ctl reconfigure
   

{{< /tab >}}

{{< tab title="Helm chart (Kubernetes)" >}}

1. Put the following content in a file named object_storage.yaml to be used as a Kubernetes Secret:

   yamlCopy

   provider: AWS
   region: us-east-1
   aws_access_key_id: <AWS_ACCESS_KEY_ID>
   aws_secret_access_key: <AWS_SECRET_ACCESS_KEY>
   

   If you're using AWS IAM profiles, omit the AWS access key and secret access key/value pairs. For example:

   yamlCopy

   provider: AWS
   region: us-east-1
   use_iam_profile: true
   

2. Create the Kubernetes Secret:

   shellCopy

   kubectl create secret generic -n <namespace> gitlab-object-storage --from-file=connection=object_storage.yaml
   

3. Export the Helm values:

   shellCopy

   helm get values gitlab > gitlab_values.yaml
   

4. Edit gitlab_values.yaml:

   yamlCopy

   global:
     appConfig:
        artifacts:
          bucket: gitlab-artifacts
        ciSecureFiles:
          bucket: gitlab-ci-secure-files
          enabled: true
        dependencyProxy:
          bucket: gitlab-dependency-proxy
          enabled: true
        externalDiffs:
          bucket: gitlab-mr-diffs
          enabled: true
        lfs:
          bucket: gitlab-lfs
        object_store:
          connection:
            secret: gitlab-object-storage
          enabled: true
          proxy_download: false
        packages:
          bucket: gitlab-packages
        terraformState:
          bucket: gitlab-terraform-state
          enabled: true
        uploads:
          bucket: gitlab-uploads
   

5. Save the file and apply the new values:

   shellCopy

   helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
   

{{< /tab >}}

{{< tab title="Docker" >}}

1. Edit docker-compose.yml:

   yamlCopy

   version: "3.6"
   services:
     gitlab:
       environment:
         GITLAB_OMNIBUS_CONFIG: |
           # Consolidated object storage configuration
           gitlab_rails['object_store']['enabled'] = true
           gitlab_rails['object_store']['proxy_download'] = false
           gitlab_rails['object_store']['connection'] = {
             'provider' => 'AWS',
             'region' => 'eu-central-1',
             'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>',
             'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>'
           }
           # OPTIONAL: The following lines are only needed if server side encryption is required
           gitlab_rails['object_store']['storage_options'] = {
             'server_side_encryption' => '<AES256 or aws:kms>',
             'server_side_encryption_kms_key_id' => '<arn:aws:kms:xxx>'
           }
           gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
           gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs'
           gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'
           gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads'
           gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
           gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
           gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
           gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
           gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
   

   If you're using AWS IAM profiles, omit the AWS access key and secret access key/value pairs. For example:

   rubyCopy

   gitlab_rails['object_store']['connection'] = {
     'provider' => 'AWS',
     'region' => 'eu-central-1',
     'use_iam_profile' => true
   }
   

2. Save the file and restart GitLab:

   shellCopy

   docker compose up -d
   

{{< /tab >}}

{{< tab title="Self-compiled (source)" >}}

1. Edit /home/git/gitlab/config/gitlab.yml and add or amend the following lines:

   yamlCopy

   production: &base
     object_store:
       enabled: true
       proxy_download: false
       connection:
         provider: AWS
         aws_access_key_id: <AWS_ACCESS_KEY_ID>
         aws_secret_access_key: <AWS_SECRET_ACCESS_KEY>
         region: eu-central-1
       storage_options:
         server_side_encryption: <AES256 or aws:kms>
         server_side_encryption_key_kms_id: <arn:aws:kms:xxx>
       objects:
         artifacts:
           bucket: gitlab-artifacts
         external_diffs:
           bucket: gitlab-mr-diffs
         lfs:
           bucket: gitlab-lfs
         uploads:
           bucket: gitlab-uploads
         packages:
           bucket: gitlab-packages
         dependency_proxy:
           bucket: gitlab-dependency-proxy
         terraform_state:
           bucket: gitlab-terraform-state
         ci_secure_files:
           bucket: gitlab-ci-secure-files
         pages:
           bucket: gitlab-pages
   

   If you're using AWS IAM profiles, omit the AWS access key and secret access key/value pairs. For example:

   yamlCopy

   connection:
     provider: AWS
     region: eu-central-1
     use_iam_profile: true
   

2. Edit /home/git/gitlab-workhorse/config.toml and add or amend the following lines:

   tomlCopy

   [object_storage]
     provider = "AWS"
   
   [object_storage.s3]
     aws_access_key_id = "<AWS_ACCESS_KEY_ID>"
     aws_secret_access_key = "<AWS_SECRET_ACCESS_KEY>"
   

   If you're using AWS IAM profiles, omit the AWS access key and secret access key/value pairs. For example:

   yamlCopy

   [object_storage.s3]
     use_iam_profile = true
   

3. Save the file and restart GitLab:

   shellCopy

   # For systems running systemd
   sudo systemctl restart gitlab.target
   
   # For systems running SysV init
   sudo service gitlab restart
   

{{< /tab >}}

{{< /tabs >}}

Migrate to object storage

To migrate existing local data to object storage see the following guides:

• Job artifacts including archived job logs
• LFS objects
• Uploads
• Merge request diffs
• Packages (optional feature)
• Dependency Proxy
• Terraform state files
• Pages content
• Project-level Secure Files

Transition to consolidated form

In storage-specific configuration:

• Object storage configuration for all types of objects such as CI/CD artifacts, LFS files, and upload attachments is configured independently.
• Object store connection parameters such as passwords and endpoint URLs is duplicated for each type.

For example, a Linux package installation might have the following configuration:

# Original object storage configuration
gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_direct_upload'] = true
gitlab_rails['artifacts_object_store_proxy_download'] = false
gitlab_rails['artifacts_object_store_remote_directory'] = 'artifacts'
gitlab_rails['artifacts_object_store_connection'] = { 'provider' => 'AWS', 'aws_access_key_id' => 'access_key', 'aws_secret_access_key' => 'secret' }
gitlab_rails['uploads_object_store_enabled'] = true
gitlab_rails['uploads_object_store_direct_upload'] = true
gitlab_rails['uploads_object_store_proxy_download'] = false
gitlab_rails['uploads_object_store_remote_directory'] = 'uploads'
gitlab_rails['uploads_object_store_connection'] = { 'provider' => 'AWS', 'aws_access_key_id' => 'access_key', 'aws_secret_access_key' => 'secret' }

Although this provides flexibility in that it makes it possible for GitLab to store objects across different cloud providers, it also creates additional complexity and unnecessary redundancy. Because both GitLab Rails and Workhorse components need access to object storage, the consolidated form avoids excessive duplication of credentials.

The consolidated form is used only if all lines from the original form is omitted. To move to the consolidated form, remove the original configuration (for example, artifacts_object_store_enabled, or uploads_object_store_connection)

Migrate objects to a different object storage provider

You may need to migrate GitLab data in object storage to a different object storage provider. The following steps show you how do this using Rclone.

The steps assume you are moving the uploads bucket, but the same process works for other buckets.

Prerequisites:

• Choose the computer to run Rclone on. Depending on how much data you are migrating, Rclone may have to run for a long time so you should avoid using a laptop or desktop computer that can go into power saving. You can use your GitLab server to run Rclone.

1. Install Rclone.

2. Configure Rclone by running the following:

   shellCopy

   rclone config
   

   The configuration process is interactive. Add at least two "remotes": one for the object storage provider your data is currently on (old), and one for the provider you are moving to (new).

3. Verify that you can read the old data. The following example refers to the uploads bucket, but your bucket may have a different name:

   shellCopy

   rclone ls old:uploads | head
   

   This should print a partial list of the objects currently stored in your uploads bucket. If you get an error, or if the list is empty, go back and update your Rclone configuration using rclone config.

4. Perform an initial copy. You do not need to take your GitLab server offline for this step.

   shellCopy

   rclone sync -P old:uploads new:uploads
   

5. After the first sync completes, use the web UI or command-line interface of your new object storage provider to verify that there are objects in the new bucket. If there are none, or if you encounter an error while running rclone sync, check your Rclone configuration and try again.

After you have done at least one successful Rclone copy from the old location to the new location, schedule maintenance and take your GitLab server offline. During your maintenance window you must do two things:

1. Perform a final rclone sync run, knowing that your users cannot add new objects so you do not leave any behind in the old bucket.
2. Update the object storage configuration of your GitLab server to use the new provider for uploads.

Alternatives to file system storage

If you're working to scale out your GitLab implementation, or add fault tolerance and redundancy, you may be looking at removing dependencies on block or network file systems. See the following additional guides:

1. Make sure the git user home directory is on local disk.
2. Configure database lookup of SSH keys to eliminate the need for a shared authorized_keys file.
3. Prevent local disk usage for job logs.
4. Disable Pages local storage.

Troubleshooting

Objects are not included in GitLab backups

As noted in the backup documentation, objects are not included in GitLab backups. You can enable backups with your object storage provider instead.

Use separate buckets

Using separate buckets for each data type is the recommended approach for GitLab. This ensures there are no collisions across the various types of data GitLab stores. Issue 292958 proposes to enable the use of a single bucket.

With Linux package and self-compiled installations, it is possible to split a single real bucket into multiple virtual buckets. If your object storage bucket is called my-gitlab-objects you can configure uploads to go into my-gitlab-objects/uploads, artifacts into my-gitlab-objects/artifacts, etc. The application acts as if these are separate buckets. Use of bucket prefixes may not work correctly with Helm backups.

Helm-based installs require separate buckets to handle backup restorations.

S3 API compatibility issues

If you encounter errors with an S3-compatible provider, see S3 compatibility and known failure modes for common causes and configuration adjustments. A 411 Length Required error in production.log is typically caused by signature streaming. Set enable_signature_v4_streaming: false to resolve it.

Artifacts always downloaded with filename download

Downloaded artifact filenames are set with the response-content-disposition header in the GetObject request. If the S3 provider does not support this header, the downloaded file is always saved as download.

Proxy Download

Clients can download files in object storage by receiving a pre-signed, time-limited URL, or by GitLab proxying the data from object storage to the client. Downloading files from object storage directly helps reduce the amount of egress traffic GitLab needs to process.

When the files are stored on local block storage or NFS, GitLab has to act as a proxy. This is not the default behavior with object storage.

The proxy_download setting controls this behavior: the default is false. Verify this in the documentation for each use case.

Set proxy_download to true if you want GitLab to proxy the files. There can be a large performance hit to the GitLab server if proxy_download is set to true. The server deployments of GitLab have proxy_download set to false.

When proxy_download to false, GitLab returns an HTTP 302 redirect with a pre-signed, time-limited object storage URL. This can result in some of the following problems:

• If GitLab is using non-secure HTTP to access the object storage, clients may generate https->http downgrade errors and refuse to process the redirect. The solution to this is for GitLab to use HTTPS. LFS, for example, generates this error:

  plaintextCopy

  LFS: lfsapi/client: refusing insecure redirect, https->http
  

• Clients need to trust the certificate authority that issued the object storage certificate, or may return common TLS errors such as:

  plaintextCopy

  x509: certificate signed by unknown authority
  

• Clients need network access to the object storage. Network firewalls could block access. Errors that might result if this access is not in place include:

  plaintextCopy

  Received status code 403 from server: Forbidden
  

• Object storage buckets need to allow Cross-Origin Resource Sharing (CORS) access from the URL of the GitLab instance. Attempting to load a PDF in the repository page may show the following error:

  plaintextCopy

  An error occurred while loading the file. Please try again later.
  

  See the LFS documentation for more details.

[!warning] Pre-signed URLs are time-limited but not tied to a specific user. Any user who obtains a pre-signed URL can access the object without authentication for the duration of the URL's validity. Direct downloads may also incur bandwidth charges between your object storage provider and the client.

ETag mismatch

Using the default GitLab settings, some S3-compatible object storage back-ends such as Alibaba might generate ETag mismatch errors.

Amazon S3 encryption

If you are seeing this ETag mismatch error with Amazon Web Services S3, it's likely this is due to encryption settings on your bucket. To fix this issue, you have two options:

• Use the consolidated form.
• Use Amazon instance profiles.

The consolidated form is recommended for S3-compatible services. Some services may also require additional server-side configuration, such as enabling a compatibility mode, to resolve ETag mismatch errors.

Without the consolidated form or instance profiles enabled, GitLab Workhorse uploads files to S3 using pre-signed URLs that do not have a Content-MD5 HTTP header computed for them. To ensure data is not corrupted, Workhorse checks that the MD5 hash of the data sent equals the ETag header returned from the S3 server. When encryption is enabled, this is not the case, which causes Workhorse to report an ETag mismatch error during an upload.

When the consolidated form is:

• Used with an S3-compatible object storage or an instance profile, Workhorse uses its internal S3 client which has S3 credentials so that it can compute the Content-MD5 header. This eliminates the need to compare ETag headers returned from the S3 server.
• Not used with an S3-compatible object storage, Workhorse falls back to using pre-signed URLs.

Google Cloud Storage encryption

{{< history >}}

• Introduced in GitLab 16.11.

{{< /history >}}

ETag mismatch errors occur also in Google Cloud Storage (GCS) when enabling data encryption with customer-managed encryption keys (CMEK).

To use CMEK, use the consolidated form.

Multi-threaded copying

GitLab uses the S3 Upload Part Copy API to accelerate copying files within a bucket. This feature is unsupported by some S3-compatible providers and they return a 404 error during upload.

To disable multi-threaded copying, ask a GitLab administrator with Rails console access to run the following command:

Feature.disable(:s3_multithreaded_uploads)

Manual testing through Rails Console

Use this approach to verify object storage connectivity when you suspect a misconfiguration. The following example tests a connection, writes a test object, and reads it back.

1. Start a Rails console.

2. Set up the object storage connection, using the same parameters you set up in /etc/gitlab/gitlab.rb, in the following example format:

   Example connection using the existing uploads configuration:

   rubyCopy

   settings = Gitlab.config.uploads.object_store.connection.deep_symbolize_keys
   connection = Fog::Storage.new(settings)
   

   Example connection using access keys:

   rubyCopy

   connection = Fog::Storage.new(
     {
       provider: 'AWS',
       region: 'eu-central-1',
       aws_access_key_id: '<AWS_ACCESS_KEY_ID>',
       aws_secret_access_key: '<AWS_SECRET_ACCESS_KEY>'
     }
   )
   

   Example connection using AWS IAM Profiles:

   rubyCopy

   connection = Fog::Storage.new(
     {
       provider: 'AWS',
       use_iam_profile: true,
       region: 'us-east-1'
     }
   )
   

3. Specify the bucket name to test against, write, and finally read a test file.

   rubyCopy

   dir = connection.directories.new(key: '<bucket-name-here>')
   f = dir.files.create(key: 'test.txt', body: 'test')
   pp f
   pp dir.files.head('test.txt')
   

Enable additional debugging

{{< history >}}

• AWS_DEBUG environment variable support introduced in GitLab 18.3.

{{< /history >}}

You can also enable additional debugging to see the HTTP requests. You should do it in the Rails Console to avoid leaking credentials in log files. The following shows how to enable request debugging for different providers:

{{< tabs >}}

{{< tab title="Amazon S3" >}}

Set the EXCON_DEBUG environment variable:

ENV['EXCON_DEBUG'] = "1"

You can also enable S3 HTTP request and response header logging in GitLab Workhorse logs by setting the AWS_DEBUG environment variable to 1. For the Linux package (Omnibus):

1. Edit /etc/gitlab/gitlab.rb and add the following lines:

   rubyCopy

   gitlab_workhorse['env'] = {
     'AWS_DEBUG' => '1'
   }
   

2. Save the file and reconfigure GitLab:

   shellCopy

   sudo gitlab-ctl reconfigure
   

   S3-compatible storage request and response headers are logged in /var/log/gitlab/gitlab-workhorse/current.

{{< /tab >}}

{{< tab title="Google Cloud Storage" >}}

Configure the logger to log to STDOUT:

Google::Apis.logger = Logger::new(STDOUT)

{{< /tab >}}

{{< tab title="Azure Blob Storage" >}}

Set the DEBUG environment variable:

ENV['DEBUG'] = "1"

{{< /tab >}}

{{< /tabs >}}

Reset the Geo tracking database to ensure full objects consistency

Assume the following Geo scenario:

• An environment consists of a Geo primary and secondary node.
• You migrated to object storage on the primary.
  • The secondary uses separate object storage buckets.
  • The option "Allow this secondary site to replicate content on Object Storage" is activated.

Such migrations can cause objects to be marked as synced in the tracking database while being physically missing from object storage. In that case, Reset your Geo secondary site replication to ensure objects state remains consistent post migration.

Inconsistencies after migrating to object storage

Data inconsistencies can occur when migrating from local to object storage. Especially in combination with Geo, when files were manually deleted prior to the migration.

For example, an instance administrator manually deletes several artifacts on the local file system. Such changes are not properly propagated to the database and result in inconsistencies. After the migration to object storage, these inconsistencies remain and can cause frictions. Geo secondaries might continue to try replicating those files as they are still referenced in the database but no longer exist.

Identify inconsistencies when using Geo

Assume the following Geo scenario:

• An environment consists of a Geo primary and secondary node
• Both systems have been migrated to object storage
  • The secondary uses the same object storage as the primary
  • The option Allow this secondary site to replicate content on Object Storage is deactivated
• Multiple uploads were manually deleted before the object storage migration
  • For this example, two images which were uploaded to an issue

In such a scenario, the secondary does no longer need to replicate any data as it uses the same object storage as the primary. Because of inconsistencies, administrators can observe that the secondary still tries to replicate data:

On the primary site:

1. In the upper-right corner, select Admin.
2. Select Geo > Sites.
3. Look at the primary site and check the verification information. All uploads were verified:
4. Look at the secondary site and check the verification information. Notice that two uploads are still being synced, even though the secondary should use the same object storage. Meaning it should not have to synchronize any uploads:

Clean up inconsistencies

[!warning] Ensure you have a recent and working backup at hand before issuing any deletion commands.

Based on the previous scenario, multiple uploads are causing inconsistencies which are used as an example below.

Proceed as follows to properly delete potential leftovers:

1. Map the identified inconsistencies to their respective model names. The model name is needed in the following steps.

   Object storage type	Model name
   Backups	not applicable
   Container registry	not applicable
   Mattermost	not applicable
   Autoscale runner caching	not applicable
   Secure Files	Ci::SecureFile
   Job artifacts	Ci::JobArtifact and Ci::PipelineArtifact
   LFS objects	LfsObject
   Uploads	Upload
   Merge request diffs	MergeRequestDiff
   Packages	Packages::PackageFile
   Dependency Proxy	DependencyProxy::Blob and DependencyProxy::Manifest
   Terraform state files	Terraform::StateVersion
   Pages content	PagesDeployment

2. Start a Rails console.

3. Query for all "files" which are still stored locally (instead of in object storage) based on the model name of the previous step. In this case, as uploads are affected, the model name Upload is used. Observe how openbao.png is still stored locally:

   rubyCopy

   Upload.with_files_stored_locally
   

   rubyCopy

   #<Upload:0x00007d35b69def68
     id: 108,
     size: 13346,
     path: "c95c1c9bf91a34f7d97346fd3fa6a7be/openbao.png",
     checksum: "db29d233de49b25d2085dcd8610bac787070e721baa8dcedba528a292b6e816b",
     model_id: 2,
     model_type: "Project",
     uploader: "FileUploader",
     created_at: Wed, 02 Apr 2025 05:56:47.941319000 UTC +00:00,
     store: 1,
     mount_point: nil,
     secret: "[FILTERED]",
     version: 2,
     uploaded_by_user_id: 1,
     organization_id: nil,
     namespace_id: nil,
     project_id: 2,
     verification_checksum: nil>]
   

4. Use the id of the identified resources to properly delete them. First, verify that it is the correct resource by using find and then run destroy:

   rubyCopy

   Upload.find(108)
   Upload.find(108).destroy
   

5. Optionally, verify that the resource was deleted correctly by running find again which should no longer find it:

   rubyCopy

   Upload.find(108)
   

   rubyCopy

   ActiveRecord::RecordNotFound: Couldn't find Upload with 'id'=108
   

Repeat the steps for all affected object storage types.

Job logs are missing in a multi-node GitLab instance

On GitLab instances with more than one Rails node (servers running the web services or Sidekiq) there needs to be a mechanism to make job logs available to all nodes after they have been sent from the Runner. Jobs logs can be stored on local disk or in object storage.

If NFS is not being used, and the incremental logging feature has not been enabled, then job logs can be lost:

1. The node which receives the log from the runner writes the log to local disk.
2. When GitLab tries to archive the log, often the job runs on a different server which cannot access the log.
3. Uploading to object storage fails.

The following error might also be logged to /var/log/gitlab/gitlab-rails/exceptions_json.log:

{
  "severity": "ERROR",
  "exception.class": "Ci::AppendBuildTraceService::TraceRangeError",
  "extra.build_id": 425187,
  "extra.body_end": 12955,
  "extra.stream_size": 720,
  "extra.stream_class": {},
  "extra.stream_range": "0-12954"
}

If CI artifacts are written to object storage in a multi-node environment, you must enable the incremental logging feature.