Database Dictionary

This page documents the database schema for GitLab, so data analysts and other groups can locate the feature categories responsible for specific database tables.

Location

Database dictionary metadata files are stored in the gitlab project under db/docs/ for the main, ci, and sec databases. For the embedding database, the dictionary files are stored under ee/db/embedding/docs/. For the geo database, the dictionary files are stored under ee/db/geo/docs/.

Example dictionary file

---
table_name: terraform_states
classes:
- Terraform::State
feature_categories:
- infrastructure_as_code
description: Represents a Terraform state backend
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26619
milestone: '13.0'
gitlab_schema: gitlab_main_org
sharding_key:
    project_id: projects
table_size: small

Adding tables

Schema

Footnotes:

1. New tables are usually small by default as they contain no data. This attribute is updated automatically monthly.

Process

When adding a table, you should:

1. Create a new file for this table in the appropriate directory:
   • gitlab_main table: db/docs/
   • gitlab_ci table: db/docs/
   • gitlab_sec table: db/docs/
   • gitlab_shared table: db/docs/
   • gitlab_embedding table: ee/db/embedding/docs/
   • gitlab_geo table: ee/db/geo/docs/
2. Name the file <table_name>.yml, and include as much information as you know about the table.
3. Include this file in the commit with the migration that creates the table.

Dropping tables

Schema

Process

When dropping a table, you should:

1. Move the dictionary file for this table to the deleted_tables directory:
   • gitlab_main table: db/docs/deleted_tables/
   • gitlab_ci table: db/docs/deleted_tables/
   • gitlab_sec table: db/docs/deleted_tables/
   • gitlab_shared table: db/docs/deleted_tables/
   • gitlab_embedding table: ee/db/embedding/docs/deleted_tables/
   • gitlab_geo table: ee/db/geo/docs/deleted_tables/
2. Add the fields removed_by_url and removed_in_milestone to the dictionary file.
3. Include this change in the commit with the migration that drops the table.

Adding views

Schema

Process

When adding a new view, you should:

1. Create a new file for this view in the appropriate directory:
   • gitlab_main view: db/docs/views/
   • gitlab_ci view: db/docs/views/
   • gitlab_sec view: db/docs/views/
   • gitlab_shared view: db/docs/views/
   • gitlab_embedding view: ee/db/embedding/docs/views/
   • gitlab_geo view: ee/db/geo/docs/views/
2. Name the file <view_name>.yml, and include as much information as you know about the view.
3. Include this file in the commit with the migration that creates the view.

Dropping views

Schema

Process

When dropping a view, you should:

1. Move the dictionary file for this table to the deleted_views directory:
   • gitlab_main view: db/docs/deleted_views/
   • gitlab_ci view: db/docs/deleted_views/
   • gitlab_sec view: db/docs/deleted_views/
   • gitlab_shared view: db/docs/deleted_views/
   • gitlab_embedding view: ee/db/embedding/docs/deleted_views/
   • gitlab_geo view: ee/db/geo/docs/deleted_views/
2. Add the fields removed_by_url and removed_in_milestone to the dictionary file.
3. Include this change in the commit with the migration that drops the view.

Sharding key fields

The sharding_key and desired_sharding_key fields track the progress of adding sharding keys to tables. These fields are used for tracking purposes and enable tooling to assist with the sharding process.

• sharding_key: Use this field only when the sharding key column has a NOT NULL constraint enforced in the database. This indicates the table is fully sharded and ready for organization isolation.
• desired_sharding_key: Use this field when the sharding key column exists but doesn't have a NOT NULL constraint yet (for example, during the backfill process). Tooling is available to make backfilling easier when a table has this field set. Once the backfill is complete and the NOT NULL constraint is added, replace desired_sharding_key with sharding_key.

For detailed instructions on adding sharding keys to tables, see the sharding guidelines.

Organization transfer support

Tables sharded by organization_id must be properly handled when transferring users or groups between organizations. The organization_transfer_support field tracks whether this support has been implemented.

When required

The organization_transfer_support field is required when a table's sharding_key includes organization_id.

Schema

organization_transfer_support: <status_value>

The value must be one of: supported, todo, or no_work_needed.

Status values

• supported: The table is properly handled in the organization transfer services.
  • Required for all new tables sharded by organization_id.
  • The table must be referenced in one of the transfer services:
    • app/services/organizations/transfer/groups_service.rb
    • app/services/organizations/transfer/users_service.rb
    • ee/app/services/ee/organizations/transfer/groups_service.rb
• todo: The table needs organization transfer support but doesn't have it yet.
  • Used for existing tables that were sharded by organization_id previously.
• no_work_needed: The table has been reviewed and does not require any updates during organization transfers.
  • Used when the table's organization_id is derived from or cascades through other relationships that are already handled.