ContextQMD
Back to Gitlabhq

GitLab generic packages repository

doc/user/packages/generic_packages/_index.md

18.11.222.9 KB
Original Source

{{< details >}}

  • Tier: Free, Premium, Ultimate
  • Offering: GitLab.com, GitLab Self-Managed, GitLab Dedicated

{{< /details >}}

Use the generic packages repository to publish and manage generic files, such as release binaries, in your project's package registry. This feature is particularly useful for storing and distributing artifacts that don't fit into specific package formats like npm or Maven.

The generic packages repository provides:

  • A place to store any file type as a package.
  • Version control for your packages.
  • Integration with GitLab CI/CD.
  • API access for automation.

Authenticate to the package registry

To interact with the package registry, you must authenticate with one of the following methods:

Do not use authentication methods other than the methods documented here. Undocumented authentication methods might be removed in the future.

When you authenticate with the package registry, you should follow these best practices:

  • To access permissions associated with the Developer role, use a personal access token.
  • Use CI/CD job tokens for automated pipelines.
  • Use deploy tokens for external system integration.
  • Always send authentication information over HTTPS.

HTTP Basic authentication

If you use a tool that doesn't support the standard authentication methods, you can use HTTP Basic authentication:

shell
curl --user "<username>:<token>" <other options> <GitLab API endpoint>

Although it is ignored, you must provide a username. The token is your personal access token, CI/CD job token, or deploy token.

Publish a package

You can publish packages with the API.

Publish a single file

To publish a single file, use the following API endpoint:

shell
PUT /projects/:id/packages/generic/:package_name/:package_version/:file_name

Replace the placeholders in the URL with your specific values:

  • :id: Your project ID or URL-encoded path
  • :package_name: Name of your package
  • :package_version: Version of your package
  • :file_name: Name of the file you're uploading. See valid package filename format below.

For example:

{{< tabs >}}

{{< tab title="Personal access token" >}}

With HTTP headers:

shell
curl --location --header "PRIVATE-TOKEN: <personal_access_token>" \
     --upload-file path/to/file.txt \
     "https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/1.0.0/file.txt"

With HTTP Basic authentication:

shell
curl --location --user "<username>:<personal_access_token>" \
     --upload-file path/to/file.txt \
     "https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/1.0.0/file.txt"

{{< /tab >}}

{{< tab title="Project access token" >}}

With HTTP headers:

shell
curl --location --header "PRIVATE-TOKEN: <project_access_token>" \
     --upload-file path/to/file.txt \
     "https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/1.0.0/file.txt"

With HTTP Basic authentication:

shell
curl --location --user "<project_access_token_username>:project_access_token" \
     --upload-file path/to/file.txt \
     "https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/1.0.0/file.txt"

{{< /tab >}}

{{< tab title="Deploy token" >}}

With HTTP headers:

shell
curl --location --header "DEPLOY-TOKEN: <deploy_token>" \
     --upload-file path/to/file.txt \
     "https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/1.0.0/file.txt"

With HTTP Basic authentication:

shell
curl --location --user "<deploy_token_username>:<deploy_token>" \
     --upload-file path/to/file.txt \
     "https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/1.0.0/file.txt"

Replace <deploy_token_username> with the username of your deploy token and <deploy_token> with your actual deploy token.

{{< /tab >}}

{{< tab title="CI/CD job token" >}}

These examples are for a .gitlab-ci.yml file. GitLab CI/CD automatically provides the CI_JOB_TOKEN.

With HTTP headers:

yaml
publish:
  stage: deploy
  script:
    - |
      curl --location --header "JOB-TOKEN: ${CI_JOB_TOKEN}" \
           --upload-file path/to/file.txt \
           "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my_package/${CI_COMMIT_TAG}/file.txt"

With HTTP Basic authentication:

yaml
publish:
  stage: deploy
  script:
    - |
      curl --location --user "gitlab-ci-token:${CI_JOB_TOKEN}" \
           --upload-file path/to/file.txt \
           "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my_package/${CI_COMMIT_TAG}/file.txt"

{{< /tab >}}

{{< /tabs >}}

Each request returns a response indicating success or failure. If your upload is successful, the response status is 201 Created.

Publish multiple files

To publish multiple files or an entire directory, you must make one API call for each file.

You should follow these best practices when you publish multiple files to the repository:

  • Versioning: Use a consistent versioning scheme for your package. This could be based on your project's version, build number, or date.
  • File organization: Consider how you want to structure your files within the package. You might want to include a manifest file that lists all the included files and their purposes.
  • Automation: Whenever possible, automate the publishing process through CI/CD pipelines. This ensures consistency and reduces manual errors.
  • Error handling: Implement error checking in your scripts. For example, check the HTTP response code from cURL to ensure each file was uploaded successfully.
  • Logging: Maintain logs of what files were uploaded, when, and by whom. This can be crucial for troubleshooting and auditing.
  • Compression: For large directories, consider compressing the contents into a single file before uploading. This can simplify the upload process and reduce the number of API calls.
  • Checksums: SHA256 checksums are automatically calculated and stored for uploaded files. Use checksums to verify the integrity of downloaded files.

For example:

{{< tabs >}}

{{< tab title="With a Bash script" >}}

Create a Bash script to iterate through files and upload them:

shell
#!/bin/bash

TOKEN="<access_token>"
PROJECT_ID="24"
PACKAGE_NAME="my_package"
PACKAGE_VERSION="1.0.0"
DIRECTORY_PATH="./files_to_upload"

for file in "$DIRECTORY_PATH"/*; do
    if [ -f "$file" ]; then
        filename=$(basename "$file")
        curl --location --header "PRIVATE-TOKEN: $TOKEN" \
             --upload-file "$file" \
             "https://gitlab.example.com/api/v4/projects/$PROJECT_ID/packages/generic/$PACKAGE_NAME/$PACKAGE_VERSION/$filename"
        echo "Uploaded: $filename"
    fi
done

{{< /tab >}}

{{< tab title="With GitLab CI/CD" >}}

For automated uploads in your CI/CD pipeline, you can iterate through your files and upload them:

yaml
upload_package:
  stage: publish
  script:
    - |
      for file in ./build/*; do
        if [ -f "$file" ]; then
          filename=$(basename "$file")
          curl --header "JOB-TOKEN: $CI_JOB_TOKEN" \
               --upload-file "$file" \
               "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my_package/${CI_COMMIT_TAG}/$filename"
          echo "Uploaded: $filename"
        fi
      done

{{< /tab >}}

{{< /tabs >}}

Maintain directory structure

To preserve the structure of a published directory, include the relative path in the file name:

shell
#!/bin/bash

TOKEN="<access_token>"
PROJECT_ID="24"
PACKAGE_NAME="my_package"
PACKAGE_VERSION="1.0.0"
DIRECTORY_PATH="./files_to_upload"

find "$DIRECTORY_PATH" -type f | while read -r file; do
    relative_path=${file#"$DIRECTORY_PATH/"}
    curl --location --header "PRIVATE-TOKEN: $TOKEN" \
         --upload-file "$file" \
         "https://gitlab.example.com/api/v4/projects/$PROJECT_ID/packages/generic/$PACKAGE_NAME/$PACKAGE_VERSION/$relative_path"
    echo "Uploaded: $relative_path"
done

Download a package

You can download packages with the API.

Download a single file

To download a single package file, use the following API endpoint:

shell
GET /projects/:id/packages/generic/:package_name/:package_version/:file_name

Replace the placeholders in the URL with your specific values:

  • :id: Your project ID or URL-encoded path
  • :package_name: Name of your package
  • :package_version: Version of your package
  • :file_name: Name of the file you're uploading

For example:

{{< tabs >}}

{{< tab title="Personal access token" >}}

With HTTP headers:

shell
curl --header "PRIVATE-TOKEN: <access_token>" \
     --location \
     "https://gitlab.example.com/api/v4/projects/1/packages/generic/my_package/0.0.1/file.txt" \
     --output file.txt

With HTTP Basic authentication:

shell
curl --user "<username>:<access_token>" \
     --location \
     "https://gitlab.example.com/api/v4/projects/1/packages/generic/my_package/0.0.1/file.txt" \
     --output file.txt

{{< /tab >}}

{{< tab title="Project access token" >}}

With HTTP headers:

shell
curl --header "PRIVATE-TOKEN: <project_access_token>" \
     --location \
     "https://gitlab.example.com/api/v4/projects/1/packages/generic/my_package/0.0.1/file.txt" \
     --output file.txt

With HTTP Basic authentication:

shell
curl --user "<project_access_token_username>:<project_access_token>" \
     --location \
     "https://gitlab.example.com/api/v4/projects/1/packages/generic/my_package/0.0.1/file.txt" \
     --output file.txt

{{< /tab >}}

{{< tab title="Deploy token" >}}

With HTTP headers:

shell
curl --header "DEPLOY-TOKEN: <deploy_token>" \
     --location \
     "https://gitlab.example.com/api/v4/projects/1/packages/generic/my_package/0.0.1/file.txt" \
     --output file.txt

With HTTP Basic authentication:

shell
curl --user "<deploy_token_username>:<deploy_token>" \
     --location \
     "https://gitlab.example.com/api/v4/projects/1/packages/generic/my_package/0.0.1/file.txt" \
     --output file.txt

{{< /tab >}}

{{< tab title="CI/CD job token" >}}

These examples are for a .gitlab-ci.yml file. GitLab CI/CD automatically provides the CI_JOB_TOKEN.

With HTTP headers:

yaml
download:
  stage: test
  script:
    - |
      curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" \
           --location \
           --output file.txt \
           "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my_package/${CI_COMMIT_TAG}/file.txt"

With HTTP Basic authentication:

yaml
download:
  stage: test
  script:
    - |
      curl --user "gitlab-ci-token:${CI_JOB_TOKEN}" \
           --location \
           --output file.txt \
           "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my_package/${CI_COMMIT_TAG}/file.txt"

Each request returns a response indicating success or failure. If your upload is successful, the response status is 201 Created.

{{< /tab >}}

{{< /tabs >}}

Download multiple files

To download multiple files or an entire directory, you must make one API call for each file, or use additional tools.

You should follow these best practices when you download multiple files from the repository:

  • Versioning: Always specify the exact version of the package you want to download to ensure consistency.
  • Directory structure: When downloading, maintain the original directory structure of the package to preserve file organization.
  • Automation: Integrate package downloads into your CI/CD pipelines or build scripts for automated workflows.
  • Error handling: Implement checks to ensure all files are downloaded successfully. You can verify the HTTP status code or check file existence after download.
  • Caching: For frequently used packages, consider implementing a caching mechanism to reduce network usage and improve build times.
  • Parallel downloads: For large packages with many files, you might want to implement parallel downloads to speed up the process.
  • Checksums: Verify the integrity of downloaded files using SHA256 checksums. Checksums are provided in response headers and API responses for verification.
  • Incremental downloads: For large packages that change frequently, consider implementing a mechanism to download only the files that have changed since the last download.

For example:

{{< tabs >}}

{{< tab title="With a Bash script" >}}

Create a bash script to download multiple files:

shell
#!/bin/bash

TOKEN="<access_token>"
PROJECT_ID="24"
PACKAGE_NAME="my_package"
PACKAGE_VERSION="1.0.0"
OUTPUT_DIR="./downloaded_files"

# Create output directory if it doesn't exist
mkdir -p "$OUTPUT_DIR"

# Array of files to download
files=("file1.txt" "file2.txt" "subdirectory/file3.txt")

for file in "${files[@]}"; do
    curl --location --header "PRIVATE-TOKEN: $TOKEN" \
         --output "$OUTPUT_DIR/$file" \
         --create-dirs \
         "https://gitlab.example.com/api/v4/projects/$PROJECT_ID/packages/generic/$PACKAGE_NAME/$PACKAGE_VERSION/$file"
    echo "Downloaded: $file"
done

{{< /tab >}}

{{< tab title="With GitLab CI/CD" >}}

For automated downloads in your CI/CD pipeline:

yaml
download_package:
  stage: build
  script:
    - |
      FILES=("file1.txt" "file2.txt" "subdirectory/file3.txt")
      for file in "${FILES[@]}"; do
        curl --location --header "JOB-TOKEN: $CI_JOB_TOKEN" \
             --output "$file" \
             --create-dirs \
             "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my_package/${CI_COMMIT_TAG}/$file"
        echo "Downloaded: $file"
      done

{{< /tab >}}

{{< /tabs >}}

Download an entire package

To download all files in a package, you must:

  1. Get the package ID.
  2. List the package contents using the GitLab API.
  3. Download each file.

Run the following command to download all files in a package:

shell
TOKEN="<access_token>"
PROJECT_ID="24"
PACKAGE_NAME="my_package"
PACKAGE_VERSION="1.0.0"
GITLAB_URL="https://gitlab.example.com"
OUTPUT_DIR="./downloaded_package"

# Create output directory
mkdir -p "$OUTPUT_DIR"

# Step 1: Get the package ID
PACKAGE_ID=$(curl --location --header "PRIVATE-TOKEN: $TOKEN" \
     "$GITLAB_URL/api/v4/projects/$PROJECT_ID/packages?package_type=generic&package_name=$PACKAGE_NAME&package_version=$PACKAGE_VERSION" \
     | jq -r ".[] | select(.name==\"$PACKAGE_NAME\" and .version==\"$PACKAGE_VERSION\") | .id")

if [ -z "$PACKAGE_ID" ] || [ "$PACKAGE_ID" = "null" ]; then
    echo "Error: Package '$PACKAGE_NAME' version '$PACKAGE_VERSION' not found"
    exit 1
fi

echo "Found package ID: $PACKAGE_ID"

# Step 2: Get list of files in the package
files=$(curl --location --header "PRIVATE-TOKEN: $TOKEN" \
     "$GITLAB_URL/api/v4/projects/$PROJECT_ID/packages/$PACKAGE_ID/package_files" \
     | jq -r '.[].file_name')

if [ -z "$files" ]; then
    echo "Error: No files found in package"
    exit 1
fi

# Step 3: Download each file
for file in $files; do
    echo "Downloading: $file"
    curl --location --header "PRIVATE-TOKEN: $TOKEN" \
         --output "$OUTPUT_DIR/$file" \
         --create-dirs \
         "$GITLAB_URL/api/v4/projects/$PROJECT_ID/packages/generic/$PACKAGE_NAME/$PACKAGE_VERSION/$file"
    # Check if download was successful
    if [ $? -eq 0 ]; then
        echo "✓ Downloaded: $file"
    else
        echo "✗ Failed to download: $file"
    fi
done

echo "Package download complete"

Delete a package

Before you delete a package, make sure you understand the associated security risks.

To delete a package, you can either:

Verify file integrity with checksums

SHA256 checksums are automatically calculated and stored for all uploaded files. You can use these checksums to verify the integrity of downloaded files.

To verify checksums, you can either:

  • Get checksums from HTTP response headers.
  • Get checksums from API responses.
  • Integrate checksum verification in CI/CD pipelines.

Get checksums from response headers

When you download files, you can get the SHA256 checksum in the X-Checksum-SHA256 response header.

Prerequisites:

To get checksums in response headers, you must either:

  • Disable object storage so files are stored locally.
  • Enable object storage, but disable direct download (files are served through GitLab Workhorse). When object storage direct download is enabled, files are redirected to the object storage provider, and custom headers like X-Checksum-SHA256 cannot be included in the redirect response.
shell
# Download a file and get the checksum from the response header
curl --header "PRIVATE-TOKEN: <access_token>" \
     --location \
     --output file.txt \
     --dump-header headers.txt \
     --url "https://gitlab.example.com/api/v4/projects/1/packages/generic/my_package/0.0.1/file.txt"

# Extract the checksum from the response header
CHECKSUM=$(grep "X-Checksum-SHA256:" headers.txt | cut -d' ' -f2 | tr -d '\r')
echo "Expected checksum: $CHECKSUM"

# Verify the downloaded file
ACTUAL_CHECKSUM=$(sha256sum file.txt | cut -d' ' -f1)
if [ "$CHECKSUM" = "$ACTUAL_CHECKSUM" ]; then
    echo "✓ File integrity verified"
else
    echo "✗ File integrity check failed"
fi

Get checksums from API responses

Get checksums by listing package files in API calls:

shell
# Get the package ID
PACKAGE_ID=$(curl --header "PRIVATE-TOKEN: <access_token>" \
     --url "https://gitlab.example.com/api/v4/projects/1/packages?package_type=generic&package_name=my_package&package_version=0.0.1" \
     | jq -r ".[] | select(.name==\"my_package\" and .version==\"0.0.1\") | .id")

# Get file information, including checksums
curl --header "PRIVATE-TOKEN: <access_token>" \
     --url "https://gitlab.example.com/api/v4/projects/1/packages/$PACKAGE_ID/package_files" \
     | jq '.[] | {file_name: .file_name, file_sha256: .file_sha256}'

Verify file integrity in CI/CD pipelines

You can integrate checksum verification into your CI/CD pipelines.

Prerequisites:

To verify checksums in CI/CD pipelines, you must either:

  • Disable object storage so files are stored locally.
  • Enable object storage, but disable direct download (files are served through GitLab Workhorse). When object storage direct download is enabled, files are redirected to the object storage provider, and custom headers like X-Checksum-SHA256 cannot be included in the redirect response.
yaml
verify_download:
  stage: test
  script:
    - |
      # Download file and get checksum from header
      curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" \
           --location \
           --output file.txt \
           --dump-header headers.txt \
           "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my_package/${CI_COMMIT_TAG}/file.txt"

      # Extract checksum from response header
      EXPECTED_CHECKSUM=$(grep "X-Checksum-SHA256:" headers.txt | cut -d' ' -f2 | tr -d '\r')

      # Calculate actual checksum
      ACTUAL_CHECKSUM=$(sha256sum file.txt | cut -d' ' -f1)

      # Verify integrity
      if [ "$EXPECTED_CHECKSUM" = "$ACTUAL_CHECKSUM" ]; then
        echo "✓ File integrity verified"
      else
        echo "✗ File integrity check failed"
        exit 1
      fi

Disable publishing duplicate package names

{{< history >}}

  • Required role changed from Developer to Maintainer in GitLab 15.0.
  • Required role changed from Maintainer to Owner in GitLab 17.0.

{{< /history >}}

By default, when you publish a package with the same name and version as an existing package, the new files are added to the existing package. You can disable publishing duplicate file names in the settings.

Prerequisites:

  • You must have the Owner role.

To disable publishing duplicate file names:

  1. In the top bar, select Search or go to and find your group.
  2. Select Settings > Packages and registries.
  3. In the Generic row of the Duplicate packages table, turn off the Allow duplicates toggle.
  4. Optional. In the Exceptions text box, enter a regular expression that matches the names and versions of packages to allow.

[!note] If Allow duplicates is turned on, you can specify package names and versions that should not have duplicates in the Exceptions text box.

Add a package retention policy

Implement a package retention policy to manage storage and maintain relevant versions.

To do so:

You can also use the API to implement custom cleanup scripts.

Generic package sample project

The Write CI-CD Variables in Pipeline project contains a working example you can use to create, upload, and download generic packages in GitLab CI/CD.

It also demonstrates how to manage a semantic version for the generic package: storing it in a CI/CD variable, retrieving it, incrementing it, and writing it back to the CI/CD variable when tests for the download work correctly.

Valid package filename format

Valid package filenames can include:

  • Letters: A-Z, a-z
  • Numbers: 0-9
  • Special characters: . (dot), _ (underscore), - (hyphen), + (plus), ~ (tilde), @ (at sign), / (forward slash)

The package filename cannot:

  • Start with a tilde (~) or the at sign (@)
  • End with a tilde (~) or the at sign (@)
  • Include spaces

Troubleshooting

HTTP 403 errors

You might get a HTTP 403 Forbidden error. This error happens when either:

  • You don't have access to a resource.
  • The package registry is not enabled for the project.

To resolve the issue, ensure the package registry is enabled, and you have permission to access it.

Internal Server error on large file uploads to S3

S3-compatible object storage limits the size of a single PUT request to 5 GB. If the aws_signature_version is set to 2 in the object storage connection settings, attempting to publish a package file larger than the 5 GB limit can result in a HTTP 500: Internal Server Error response.

If you are receiving HTTP 500: Internal Server Error responses when publishing large files to S3, set the aws_signature_version to 4:

ruby
# Consolidated Object Storage settings
gitlab_rails['object_store']['connection'] = {
  # Other connection settings
  'aws_signature_version' => '4'
}
# OR
# Storage-specific form settings
gitlab_rails['packages_object_store_connection'] = {
  # Other connection settings
  'aws_signature_version' => '4'
}