ContextQMD
Back to Gitlabhq

Configure OpenID Connect in AWS to retrieve temporary credentials

doc/ci/cloud_services/aws/_index.md

18.11.212.9 KB
Original Source

{{< details >}}

  • Tier: Free, Premium, Ultimate
  • Offering: GitLab.com, GitLab Self-Managed, GitLab Dedicated

{{< /details >}}

[!warning] CI_JOB_JWT_V2 was deprecated in GitLab 15.9 and is scheduled to be removed in GitLab 17.0. Use ID tokens instead.

This tutorial shows you how to use a GitLab CI/CD job with a JSON web token (JWT) to retrieve temporary credentials from AWS without storing secrets. To do this, you must configure OpenID Connect (OIDC) for ID federation between GitLab and AWS. For background and requirements for integrating GitLab using OIDC, see Connect to cloud services.

To complete this tutorial:

  1. Add the identity provider
  2. Configure the role and trust
  3. Retrieve a temporary credential

Add the identity provider

Create GitLab as a IAM OIDC provider in AWS following these instructions.

Include the following information:

  • Provider URL: The address of your GitLab instance, such as https://gitlab.com or http://gitlab.example.com. This address must be publicly accessible. If this is not publicly available, see how to configure a non-public GitLab instance

  • Audience: The logical name of the target service you intend to use the requested security token with.

    • In AWS OIDC integrations, this typically matches the audience value configured in your IAM OIDC identity provider (often sts.amazonaws.com or your GitLab instance URL).
    • This value is validated by AWS to ensure the token was intended for your specific identity provider.

    [!note] Using https://gitlab.com or your GitLab instance URL might work if the AWS identity provider reference matches it, but this is semantically misleading. The audience should represent the service that validates and accepts the token.

Configure a role and trust

After you create the identity provider, configure a web identity role with conditions for limiting access to GitLab resources. Temporary credentials are obtained using AWS Security Token Service, so set the Action to sts:AssumeRoleWithWebIdentity.

You can create a custom trust policy for the role to limit authorization to a specific group, project, branch, or tag. For the full list of supported filtering types, see Connect to cloud services.

json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::AWS_ACCOUNT:oidc-provider/gitlab.example.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "gitlab.example.com:sub": "project_path:mygroup/myproject:ref_type:branch:ref:main"
        }
      }
    }
  ]
}

After the role is created, attach a policy defining permissions to an AWS service (S3, EC2, Secrets Manager).

Retrieve temporary credentials

After you configure the OIDC and role, the GitLab CI/CD job can retrieve a temporary credential from AWS Security Token Service (STS).

yaml
assume role:
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.example.com
  script:
    # this is split out for correct exit code handling
    - >
      aws_sts_output=$(aws sts assume-role-with-web-identity
      --role-arn ${ROLE_ARN}
      --role-session-name "GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
      --web-identity-token ${GITLAB_OIDC_TOKEN}
      --duration-seconds 3600
      --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
      --output text)
    - export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" $aws_sts_output)
    - aws sts get-caller-identity
  • ROLE_ARN: The role ARN defined in this step.
  • GITLAB_OIDC_TOKEN: An OIDC ID token.

Working examples

Configure a non-public GitLab instance

{{< details >}}

  • Tier: Free, Premium, Ultimate
  • Offering: GitLab Self-Managed

{{< /details >}}

{{< history >}}

{{< /history >}}

[!warning] This workaround is an advanced configuration option with security considerations to understand. You must be careful to correctly sync the OpenID configuration and the public keys from your private GitLab Self-Managed instance to a publicly available location such as an S3 bucket. You must also ensure that the S3 bucket and files inside are properly secured. Failing to properly secure the S3 bucket could lead to the takeover of any cloud accounts associated with this OpenID Connect identity.

If your GitLab instance is not publicly accessible, configuring OpenID Connect in AWS is not possible by default. You can use a workaround to make some specific configuration publicly accessible, enabling OpenID Connect configuration for the instance:

  1. Store authentication details for your GitLab instance at a publicly available location, for example in S3 files:

    • Host the OpenID configuration for your instance in an S3 file. The configuration is available at /.well-known/openid-configuration, like http://gitlab.example.com/.well-known/openid-configuration. Update the issuer: and jwks_uri: values in the configuration file to point to the publicly available locations.
    • Host the public keys for your instance URL in an S3 file. The keys are available at available at /oauth/discovery/keys, like http://gitlab.example.com/oauth/discovery/keys.

    For example:

    • OpenID configuration file: https://example-oidc-configuration-s3-bucket.s3.eu-north-1.amazonaws.com/.well-known/openid-configuration.
    • JWKS (JSON Web Key Sets): https://example-oidc-configuration-s3-bucket.s3.eu-north-1.amazonaws.com/oauth/discovery/keys.
    • The issuer claim iss: in the ID Tokens and the issuer: value in the OpenID configuration would be: https://example-oidc-configuration-s3-bucket.s3.eu-north-1.amazonaws.com

  2. Optional. Use an OpenID configuration validator like the OpenID Configuration Endpoint Validator to validate your publicly available OpenID configuration.

  3. Configure a custom issuer claim for your ID tokens. By default, GitLab ID tokens have the issuer claim iss: set as the address of your GitLab instance, for example: http://gitlab.example.com.

  4. Update the issuer URL:

    {{< tabs >}}

    {{< tab title="Linux package (Omnibus)" >}}

    1. Edit /etc/gitlab/gitlab.rb:

      ruby
      gitlab_rails['ci_id_tokens_issuer_url'] = '<public_url_with_openid_configuration_and_keys>'

      Replace <public_url_with_openid_configuration_and_keys> with a URL like https://example-oidc-configuration-s3-bucket.s3.eu-north-1.amazonaws.com.

    2. Save the file and reconfigure GitLab for the changes to take effect.

    {{< /tab >}}

    {{< tab title="Helm chart (Kubernetes)" >}}

    1. Export the Helm values:

      shell
      helm get values gitlab > gitlab_values.yaml

    2. Edit gitlab_values.yaml:

      yaml
      global:
  appConfig:
    ciIdTokens:
      issuerUrl: '<public_url_with_openid_configuration_and_keys>'

      Replace <public_url_with_openid_configuration_and_keys> with a URL like https://example-oidc-configuration-s3-bucket.s3.eu-north-1.amazonaws.com.

    3. Save the file and apply the new values:

      shell
      helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab

    {{< /tab >}}

    {{< tab title="Docker" >}}

    1. Edit docker-compose.yml:

      yaml
      version: "3.6"
services:
  gitlab:
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        gitlab_rails['ci_id_tokens_issuer_url'] = '<public_url_with_openid_configuration_and_keys>'

      Replace <public_url_with_openid_configuration_and_keys> with a URL like https://example-oidc-configuration-s3-bucket.s3.eu-north-1.amazonaws.com.

    2. Save the file and restart GitLab:

      shell
      docker compose up -d

    {{< /tab >}}

    {{< tab title="Self-compiled (source)" >}}

    1. Edit /home/git/gitlab/config/gitlab.yml:

      yaml
       production: &base
   ci_id_tokens:
     issuer_url: '<public_url_with_openid_configuration_and_keys>'

      Replace <public_url_with_openid_configuration_and_keys> with a URL like https://example-oidc-configuration-s3-bucket.s3.eu-north-1.amazonaws.com.

    2. Save the file and reconfigure GitLab for the changes to take effect.

    {{< /tab >}}

    {{< /tabs >}}

  5. Run the ci:validate_id_token_configuration Rake task to validate the CI/CD ID token configuration.

Troubleshooting

Error: Not authorized to perform sts:AssumeRoleWithWebIdentity

If you see this error:

plaintext
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation:
Not authorized to perform sts:AssumeRoleWithWebIdentity

It can occur for multiple reasons:

  • The cloud administrator has not configured the project to use OIDC with GitLab.
  • The role is restricted from being run on the branch or tag. See configure a conditional role.
  • StringEquals is used instead of StringLike when using a wildcard condition. See related issue.

Could not connect to openid configuration of provider error

After adding the Identity Provider in AWS IAM, you might get the following error:

plaintext
Your request has a problem. Please see the following details.
  - Could not connect to openid configuration of provider: `https://gitlab.example.com`

This error occurs when the OIDC identity provider's issuer presents a certificate chain that's out of order, or includes duplicate or additional certificates.

Verify your GitLab instance's certificate chain. The chain must start with the domain or issuer URL, then the intermediate certificate, and end with the root certificate. Use this command to review the certificate chain, replacing gitlab.example.com with your GitLab hostname:

shell
echo | /opt/gitlab/embedded/bin/openssl s_client -connect gitlab.example.com:443

Couldn't retrieve verification key from your identity provider error

You might receive an error similar to:

  • An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements

This error might be because:

  • The .well_known URL and jwks_uri of the identity provider (IdP) are inaccessible from the public internet.
  • A custom firewall is blocking the requests.
  • There's latency of more than 5 seconds in API requests from the IdP to reach the AWS STS endpoint.
  • STS is making too many requests to your .well_known URL or the jwks_uri of the IdP.

As documented in the AWS Knowledge Center article for this error, your GitLab instance needs to be publicly accessible so that the .well_known URL and jwks_uri can be resolved. If this is not possible, for example if your GitLab instance is in an offline environment, see how to configure a non-public GitLab instance