docs/platform/connector-development/connector-builder-ui/authentication.md
Authentication allows the connector to check whether it has sufficient permission to fetch data and communicate its identity to the API. The authentication feature provides a secure way to configure authentication using a variety of methods.
Credentials, like a username and password, aren't specified as part of the connector. Instead, your connector's end user inputs them as part of their configuration when they set up the connector. During development, it's possible to provide testing credentials, but those aren't saved along with the connector. Airbyte stores credentials that are part of the source configuration in a secure way, while it saves connector configurations in the regular database.
In the "Authentication" section on the "Global Configuration" page in the connector builder, the authentication method can be specified. This configuration is shared for all streams - it's not possible to use different authentication methods for different streams in the same connector. In case your API uses multiple or custom authentication methods, you can use the low-code CDK or Python CDK.
If your API doesn't need authentication, set it to use "No auth". This means the connector can make requests to the API without providing any credentials, which might be the case for some public open APIs or private APIs only available in local networks.
Check the documentation of the API you want to integrate for the authentication method you can use. The Connector Builder supports the following ones.
Select the matching authentication method for your API and check the sections below for more information about individual methods.
If requests are authenticated using the Basic HTTP authentication method, the documentation page will likely contain one of the following keywords:
The Basic HTTP authentication method is a standard and doesn't require any further configuration. Username and password are set via "Testing values" in the connector builder and by the end user when configuring this connector as a Source.
The Greenhouse API is an API using basic authentication. Sometimes, only a username and no password is required, like for the Chargebee API. In these cases, just leave the password input empty.
In the basic authentication scheme, the supplied username and password are concatenated with a colon : and encoded using the base64 algorithm. For username user and password passwd, the base64-encoding of user:passwd is dXNlcjpwYXNzd2Q=.
When fetching records, this string is sent as part of the Authorization header:
curl -X GET \
-H "Authorization: Basic dXNlcjpwYXNzd2Q=" \
https://harvest.greenhouse.io/v1/<stream path>
If requests are authenticated using Bearer authentication, the documentation will probably mention "bearer token" or "token authentication". In this scheme, the Authorization header of the HTTP request is set to Bearer <token>.
Like the Basic HTTP authentication it does not require further configuration. The bearer token can be set via "Testing values" in the connector builder and by the end user when configuring this connector as a source.
The Sendgrid API and the Square API support Bearer authentication.
When fetching records, the token is sent along as the Authorization header:
curl -X GET \
-H "Authorization: Bearer <bearer token>" \
https://api.sendgrid.com/<stream path>
The API key authentication method is similar to the Bearer authentication but allows you to configure where to inject the API key (header, request param or request body), as well as under which field name. The used injection mechanism and the field name is part of the connector definition while the API key itself can be set via "Testing values" in the connector builder as well as when configuring this connector as a Source.
The following table helps with which mechanism to use for which API:
| Description | Injection mechanism |
|---|---|
| (HTTP) header | header |
| Query parameter / query string / request parameter / URL parameter | request_parameter |
| Form encoded request body / form data | body_data |
| JSON encoded request body | body_json |
The CoinAPI.io API uses API key authentication via the X-CoinAPI-Key header.
When fetching records, Airbyte includes the api token in the request using the configured header:
curl -X GET \
-H "X-CoinAPI-Key: <api-key>" \
https://rest.coinapi.io/v1/<stream path>
In this case the injection mechanism is header and the field name is X-CoinAPI-Key.
If requests are authenticated using JWT (JSON Web Token) authentication, the API documentation will likely mention one of the following:
JWT authentication generates a signed token using a secret key and algorithm, which is then sent with each request. The connector builder supports various JWT algorithms and configuration options.
The JWT authentication method requires the following configuration:
Secret Key - The secret key used to sign the JWT token. This can be set via "Testing values" in the connector builder and by the end user when configuring this connector as a source.
Algorithm - The algorithm used to sign the token.
Header Prefix - The prefix to use in the Authorization header (defaults to "Bearer")
JWT Headers - Additional headers to include in the JWT header object
JWT Payload - Additional properties to include in the JWT payload
Base64 Encode Secret Key - When enabled, the secret key will be base64 encoded before being used to sign the JWT. Only enable this if required by the API.
Token Duration - The amount of time in seconds a JWT token can be valid after being issued.
Header Prefix - The prefix to be used within the Authentication header, like Bearer or Basic.
Additional JWT Headers - Extra headers to include with the JWT headers object
Additional JWT Payload - Extra properties to add to the JWT payload
When fetching records, the JWT token is generated and sent as part of the Authorization header:
curl -X GET \
-H "Authorization: Bearer <jwt-token>" \
https://api.example.com/<stream path>
The JWT token is automatically generated by the connector using the configured secret key and algorithm, and includes any additional headers or payload properties you've specified.
OAuth 2 allows for authorization without a user sharing their credentials with Airbyte. It's best to implement OAuth by enabling "Declarative OAuth Connector Specification". If the API doesn't support this, you can use the legacy approach without it, but this involves more work for you and your connector's users.
The best way to implement OAuth is to enable Declarative OAuth Connector Specification. This allows Airbyte connectors to support the full authorization flow and request user consent, handles token refresh automatically, and allows for injection of custom token headers and query parameters.
When a user authenticates, they'll be redirected to the consent URL to grant permissions, then the connector automatically exchanges the authorization code for an access token.
However, some APIs might not support this, or you might already have OAuth credentials. In these cases, turn this option off.
Consent URL: Defines how to construct the URL where users grant consent. It supports:
https://api.example.com/oauth/authorize?client_id={{client_id_value}}&redirect_uri={{redirect_uri_value}}&scope={{scope_value}}&state={{state_value}}Access Token URL: Specifies how to exchange the authorization code for an access token. It supports:
https://api.example.com/oauth/tokenYou can also configure these optional fields if they're required by the source.
Access Token Headers: The DeclarativeOAuth Specific optional headers to inject while exchanging the auth_code to access_token during completeOAuthFlow step.
Access Token Query Params (Json Encoded): The DeclarativeOAuth Specific optional query parameters to inject while exchanging the auth_code to access_token during completeOAuthFlow step. When this property is provided, the query params will be encoded as Json and included in the outgoing API request.
Configurable State Query Param: The DeclarativeOAuth Specific object to provide the criteria of how the state query param should be constructed, including length and complexity.
Client ID Key Override: The DeclarativeOAuth Specific optional override to provide the custom client_id key name, if required by data-provider.
Client Secret Key Override: The DeclarativeOAuth Specific optional override to provide the custom scope key name, if required by data-provider.
Scopes Key Override: The DeclarativeOAuth Specific optional override to provide the custom scope key name, if required by data-provider.
State Key Override: The DeclarativeOAuth Specific optional override to provide the custom state key name, if required by data-provider.
Auth Code Key Override: The DeclarativeOAuth Specific optional override to provide the custom code key name to something like auth_code or custom_auth_code, if required by data-provider.
Redirect URI Key Override: The DeclarativeOAuth Specific optional override to provide the custom redirect_uri key name to something like callback_uri, if required by data-provider.
This legacy approach allows you to use an existing access token or refresh token to authenticate with the API. This method is useful when you already have OAuth credentials or when the API doesn't support the full OAuth 2.0 authorization code flow.
This method requires you to manually obtain the initial OAuth tokens outside of Airbyte. You'll need to implement the initial OAuth flow yourself to get the access_token and refresh_token, then provide them to the connector as part of the configuration.
This method doesn't provide a single-click authentication experience for end users. Users still need to manually obtain the client ID, client secret, and refresh token from the API and enter them into the configuration form.
This method implements authentication using OAuth 2.0 with refresh token grant type and client credentials grant type.
In this approach, the OAuth endpoint of an API is called with client ID and client secret and/or a long-lived refresh token that you provide when configuring the connector as a Source. These credentials are used to obtain a short-lived access token that's used for making requests to extract records. If the access token expires, the connector will automatically request a new one.
The connector needs to be configured with the endpoint to call to obtain access tokens using the client ID/secret and/or the refresh token. OAuth client ID/secret and the refresh token are provided via "Testing values" in the connector builder as well as when configuring this connector as a Source.
Depending on how the refresh endpoint is implemented, additional configuration might be necessary to specify how to request an access token with the right permissions and how to extract the access token and expiry date from the response.
If the API uses other grant types (like PKCE), you may need to use the low-code CDK or Python CDK instead.
You can also configure also a number of optional fields, if the API requires them.
access_tokenexpires_inThe Square API supports OAuth.
In this case, the authentication method has to be configured like this:
https://connect.squareup.com/oauth2/tokenexpires_atWhen running a sync, the connector first sends client id, client secret and refresh token to the token refresh endpoint:
curl -X POST \
-H "Content-Type: application/json" \
-d '{"client_id": "<client id>", "client_secret": "<client secret>", "refresh_token": "<refresh token>", "grant_type": "refresh_token" }' \
<token refresh endpoint>
The response is a JSON object containing an access_token property and an expires_at property:
{"access_token":"<access-token>", "expires_at": "2023-12-12T00:00:00"}
The expires_at date tells the connector how long the access token can be used - if this point in time is passed, a new access token is requested automatically.
When fetching records, the access token is sent along as part of the Authorization header:
curl -X GET \
-H "Authorization: Bearer <access-token>" \
https://connect.squareup.com/v2/<stream path>
In most cases, OAuth refresh tokens are long-lived and can be reused to create access tokens for every sync. However, some APIs invalidate the refresh token after each use and return a new refresh token along with the access token. One example of this behavior is the Smartsheets API.
For APIs that use single-use refresh tokens, turn on Refresh Token Updater. In this case:
refresh_token is used to extract the new refresh token, but this can be configured using the "Refresh token property name" settingThis ensures that the connector can continue to authenticate successfully even when refresh tokens are single-use.
Some APIs require callers to first fetch a unique token from one endpoint, then make the rest of their calls to all other endpoints using that token to authenticate themselves. These tokens usually have an expiration time, after which a new token needs to be re-fetched to continue making requests. This flow can be achieved through using the Session Token Authenticator.
If requests are authenticated using the Session Token authentication method, the API documentation page will likely contain one of the following keywords:
The configuration of a Session Token authenticator is a bit more involved than other authenticators, as you need to configure both how to make requests to the session token retrieval endpoint (which requires its own authentication method), as well as how the token is extracted from that response and used for the data requests.
We will walk through each part of the configuration below. Throughout this, we will refer to the Metabase API as an example of an API that uses session token authentication.
Session Token Retrieval - This is a group of fields which configures how the session token is fetched from the session token endpoint in your API. Once the session token is retrieved, your connector will reuse that token until it expires, at which point it will retrieve a new session token using this configuration.
URL - the full URL of the session token endpoint
https://<app_name>.metabaseapp.com/api/session.HTTP Method - the HTTP method that should be used when retrieving the session token endpoint, either GET or POST
POST for its /api/session requests.Authentication Method - configures the method of authentication to use for the session token retrieval request only
/api/session endpoint takes in a username and password in the request body. Since this is a non-standard authentication method, we must set this inner Authentication Method to No Auth, and instead configure the Request Body to pass these credentials (discussed below).Query Parameters - used to attach query parameters to the session token retrieval request
/api/session request, so this is left unset.Request Headers - used to attach headers to the session token retrieval request
/api/session request, so this is left unset.Request Body - used to attach a request body to the session token retrieval request
JSON (key-value pairs) here and set the username and password fields (using User Inputs for the values to make the connector reusable), so this would end up looking like:
username, Value: {{ config['username'] }}password, Value: {{ config['password'] }}Error Handler - used to handle errors encountered when retrieving the session token
Session Token Path - An array of values to form a path into the session token retrieval response which points to the session token value
/api/session response looks like {"id":"<session-token-value>"}, so the value here would simply be id.Expiration Duration - An ISO 8601 duration indicating how long the session token has until it expires
Expiration Duration field to the expected minimum duration to avoid problems during syncing./api/session endpoint expires after 14 days by default, so this value can be set to P2W or P14D.Data Request Authentication - Configures how the session token is used to authenticate the data requests made to the API
API Key if your session token needs to be injected into a query parameter or header of the data requests.
API Key, Inject Session Token into outgoing HTTP Request would be set to Header, and Header Name would be set to X-Metabase-Session.Bearer if your session token needs to be sent as a standard Bearer token.Selective Authenticator allows you to dynamically choose between different authentication methods based on a configuration property value. This is useful when your connector needs to support multiple authentication options and let users select which one to use.
This authentication method isn't well-supported in the UI because you can't define nested inputs on the Inputs form. If you need to use this authentication method, switch to YAML mode and configure the spec with nested inputs.
Shared Authenticator is a UI feature that allows you to share authenticator configurations across different parts of your connector. This helps reduce duplication and ensures consistency when the same authentication method is used in multiple places.
In the Connector Builder UI, you'll see a link icon with a toggle switch next to authenticator fields that can be shared. This feature allows you to:
If your connector uses the same API key authentication for both data retrieval and metadata requests, you can configure the API key once and share it across both components using the Shared Authenticator feature.
Some APIs require complex custom authentication schemes involving signing requests or doing multiple requests to authenticate. In these cases, use the low-code/YAML mode or Python CDK.