docusaurus/platform_versioned_docs/version-2.0/enterprise-setup/chart-v2-enterprise.mdx
import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import EnvironmentVarConversion from '@site/static/_extraenv_to_helm_chart_v2.md'; import HelmSyntaxConversion from '@site/static/_helm-chart-2-syntax-changes.md';
Airbyte has upgraded its Helm chart to a new version called "V2." Upgrading to Helm chart V2 is currently optional. At some future date the V2 Helm chart will become the standard, so we advise that you upgrade your existing deployment to use the new chart before the transition. If you're a new Airbyte customer, you can skip the upgrade altogether and start with the new chart.
:::important Open Source Users Follow the Core guide instead. :::
Upgrading to the new Helm chart now has the following benefits.
By upgrading in advance, you can schedule this upgrade for a convenient time. Avoid blocking yourself from upgrading Airbyte to a future version when the new chart is mandatory and you're busy.
The new Helm chart doesn't require Keycloak. If you don't want to use Keycloak for authentication, or want to use generic OIDC, you must run Helm chart V2.
The new Helm chart is more aligned with Helm's best practices for chart design.
The new Helm chart has broader and more detailed options to customize your deployment. In most cases, it's no longer necessary to specify environment variables in your values.yaml file because the chart offers a more detailed interface for customization. If you do need to use environment variables, you can use fewer of them.
The following versions of Airbyte can use Helm chart V2:
The following versions of Airbyte can't use Helm chart V2:
Airbyte versions before 1.6.0
Airbyte versions installed and managed with abctl
:::important "It's dangerous to go alone! Take this." Engage your Airbyte Solution Architect for help with this migration. We caution against doing it alone. Airbyte can provide guidance to help you manage uncommon customizations and verify the migration is successful. :::
In most cases, upgrading is straightforward. To upgrade to Helm chart V2, you complete the following steps.
Ensure you have configured Airbyte to use an external database and external bucket storage
Prepare to deploy a fresh installation of Airbyte.
Create a new values.yaml file.
Deploy a new version of Airbyte using your new values.yaml file and the new Helm chart version.
Airbyte's solutions team guides Enterprise customers to configure your own external database and external storage, as explained in the implementation guide. Verify that you've set these up, but you almost certainly have.
When moving to Helm chart V2, deploy Airbyte with a new namespace and use a fresh values and secrets file. It is possible to do a straight upgrade, but different Airbyte users have different and sometimes complex configurations that could produce unique and unexpected situations during the upgrade. By doing a fresh install, you create a separate environment that's easier to troubleshoot if something in your values or secrets files acts unexpectedly.
kubectl create namespace airbyte-v2
Helm chart V2 uses a different repo URL (/charts) than V1 did (/helm-charts). In your command line tool, add this repo and index it.
helm repo add airbyte-v2 https://airbytehq.github.io/charts
helm repo update
You can browse all charts uploaded to your repository.
helm search repo airbyte-v2
In most cases, the adjustments to values.yaml are small and involve changing keys and moving sections. This section walks you through the main updates you need to make. If you already know what to do, see Values.yaml reference for the full V1 and V2 interfaces.
Airbyte recommends approaching this project in this way:
Note the customizations in your V1 values.yaml file to ensure you don't forget anything.
Start with a basic V2 values.yaml to verify that it works. Map your V1 settings to V2, transferring one set of configurations at a time.
Don't test in production.
Follow the steps below to start generating values.yaml.
Create a new values.yaml file on your machine. In that file, create your basic global configuration.
global:
edition: enterprise
enterprise:
secretName: "" # Secret name where an Airbyte license key is stored
licenseKeySecretKey: "" # The key within `licenseKeySecretName` where the Airbyte license key is stored
airbyteUrl: "" # The URL where Airbyte will be reached; This should match your Ingress host
Optional: deploy Airbyte before you add additional configurations. If there are issues with your deployment, troubleshooting them is easier before you integrate additional services.
</details> <details> <summary> Add `auth` and single sign on </summary>You can implement single sign on (SSO) with OIDC or new generic OIDC. For more help, see Single sign on (SSO).
<Tabs> <TabItem value="oidc" label="OIDC">global:
auth:
# -- Admin user configuration
instanceAdmin:
firstName: ""
lastName: ""
emailSecretKey: "" # The key within `emailSecretName` where the initial user's email is stored
passwordSecretKey: "" # The key within `passwordSecretName` where the initial user's password is stored
# -- SSO Identify Provider configuration; (requires Enterprise)
identityProvider:
secretName: "" # Secret name where the OIDC configuration is stored
type: "oidc"
oidc:
# -- OIDC application domain
domain: ""
# -- OIDC application name
appName: ""
# -- The key within `clientIdSecretName` where the OIDC client id is stored
clientIdSecretKey: ""
# -- The key within `clientSecretSecretName` where the OIDC client secret is stored
clientSecretSecretKey: ""
global:
auth:
# -- Admin user configuration
instanceAdmin:
firstName: ""
lastName: ""
emailSecretKey: "" # The key within `emailSecretName` where the initial user's email is stored
passwordSecretKey: "" # The key within `passwordSecretName` where the initial user's password is stored
# -- SSO Identify Provider configuration; (requires Enterprise)
identityProvider:
secretName: "" # Secret name where the OIDC configuration is stored
type: "generic-oidc"
genericOidc:
clientId: ""
audience: ""
extraScopes: ""
issuer: ""
endpoints:
authorizationServerEndpoint: ""
jwksEndpoint: ""
fields:
subject: sub
email: email
name: name
issuer: iss
Disable Airbyte's default Postgres database and add your own. The main difference in Helm chart V2 is the global.database.database key has changed to global.database.name.
global:
database:
# -- Secret name where database credentials are stored
secretName: "" # e.g. "airbyte-config-secrets"
# -- The database host
host: ""
# -- The database port
port:
# -- The database name - this key used to be "database" in Helm chart 1.0
name: ""
# Use EITHER user or userSecretKey, but not both
# -- The database user
user: ""
# -- The key within `secretName` where the user is stored
userSecretKey: "" # e.g. "database-user"
# Use EITHER password or passwordSecretKey, but not both
# -- The database password
password: ""
# -- The key within `secretName` where the password is stored
passwordSecretKey: "" # e.g."database-password"
postgresql:
enabled: false
global:
storage:
secretName: ""
type: minio # default storage is minio. Set to s3, gcs, or azure, according to what you use.
bucket:
log: airbyte-bucket
state: airbyte-bucket
workloadOutput: airbyte-bucket
activityPayload: airbyte-bucket
# Set ONE OF the following storage types, according to your specification above
# S3
s3:
region: "" ## e.g. us-east-1
authenticationType: credentials ## Use "credentials" or "instanceProfile"
accessKeyId: ""
secretAccessKey: ""
# GCS
gcs:
projectId: <project-id>
credentialsJson: <base64-encoded>
credentialsJsonPath: /secrets/gcs-log-creds/gcp.json
# Azure
azure:
# one of the following: connectionString, connectionStringSecretKey
connectionString: <azure storage connection string>
connectionStringSecretKey: <secret coordinate containing an existing connection-string secret>
global:
secretsManager:
enabled: false
type: "" # one of: VAULT, GOOGLE_SECRET_MANAGER, AWS_SECRET_MANAGER, AZURE_KEY_VAULT, TESTING_CONFIG_DB_TABLE
secretName: "airbyte-config-secrets"
# Set ONE OF the following groups of configurations, based on your configuration in global.secretsManager.type.
awsSecretManager:
region: <aws-region>
authenticationType: credentials ## Use "credentials" or "instanceProfile"
tags: ## Optional - You may add tags to new secrets created by Airbyte.
- key: ## e.g. team
value: ## e.g. deployments
- key: business-unit
value: engineering
kms: ## Optional - ARN for KMS Decryption.
# OR
googleSecretManager:
projectId: <project-id>
credentialsSecretKey: gcp.json
# OR
azureKeyVault:
tenantId: ""
vaultUrl: ""
clientId: ""
clientIdSecretKey: ""
clientSecret: ""
clientSecretSecretKey: ""
tags: ""
# OR
vault:
address: ""
prefix: ""
authToken: ""
authTokenSecretKey: ""
If you're using version 1.7 or later, you can enable audit logging. Unlike Helm chart V1, it's no longer necessary to specify environment variables. For more help with audit logging, see Audit logging.
server:
auditLogginEnabled: true
storage:
bucket:
auditLogging: your-audit-logging-bucket-name-here
If you have further customizations in your V1 values.yaml file, move those over to your new values.yaml file, and update key names where appropriate.
Change hyphenated V1 keys keys to camel case in V2. For example, when copying over workload-launcher, change it to workloadLauncher.
Some keys have different names. For example, orchestrator is containerOrchestrator in V2.
Here is the full list of changes.
<HelmSyntaxConversion/> </details> <details> <summary> Convert `extraEnv` variables </summary>In previous versions of your values.yaml file, you might have specified a number of environment variables through extraEnv. Many (but not all) of these variables have a dedicated interface in Helm chart V2. For example, look at the following configuration, which tells workload-launcher to run pods in the jobs node group.
workload-launcher:
nodeSelector:
type: static
## Pods spun up by the workload launcher will run in the 'jobs' node group.
extraEnv:
- name: JOB_KUBE_NODE_SELECTORS
value: type=jobs
- name: SPEC_JOB_KUBE_NODE_SELECTORS
value: type=jobs
- name: CHECK_JOB_KUBE_NODE_SELECTORS
value: type=jobs
- name: DISCOVER_JOB_KUBE_NODE_SELECTORS
value: type=jobs
You can specify these values directly without using environment variables, achieving the same effect.
global:
jobs:
kube:
nodeSelector:
type: jobs
scheduling:
check:
nodeSelectors:
type: jobs
discover:
nodeSelectors:
type: jobs
spec:
nodeSelectors:
type: jobs
workloadLauncher:
nodeSelector:
type: static
Identify the Helm chart version that corresponds to the platform version you want to run. Most Helm chart versions are designed to work with one Airbyte version, and they don't necessarily have the same version number.
helm search repo airbyte-v2 --versions
You should see something like this:
NAME CHART VERSION APP VERSION DESCRIPTION
airbyte-v2/airbyte 2.0.18 2.0.0 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.17 1.8.5 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.16 1.8.4 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.15 1.8.4 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.14 1.8.4 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.13 1.8.3 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.12 1.8.2 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.11 1.8.2 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.10 1.8.1 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.9 1.8.0 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.8 1.8.0 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.7 1.7.1 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.6 1.7.1 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.5 1.7.0 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.4 1.6.3 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.3 1.6.2 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.2 1.6.2 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.1 1.6.1 Helm chart to deploy airbyte
airbyte-v2/airbyte 2.0.0 1.6.0 Helm chart to deploy airbyte
airbyte-v2/airbyte-data-plane 2.0.0 2.0.0 A Helm chart for installing an Airbyte Data Plane.
Install Airbyte into your Helm chart V2 namespace. In this example, you install Airbyte version 2.0.
helm install airbyte airbyte-v2/airbyte \
--namespace airbyte-v2 \ # Target Kubernetes namespace
--values ./values.yaml \ # Custom configuration values
--version 2.0.18 # Helm chart version to use