Airbyte CI CLI

What is it?

airbyte-ci is a command line interface to run CI/CD pipelines. The goal of this CLI is to offer developers a tool to run these pipelines locally and in a CI context with the same guarantee. It can prevent unnecessary commit -> push cycles developers typically go through when they when to test their changes against a remote CI. This is made possible thanks to the use of Dagger, a CI/CD engine relying on Docker Buildkit to provide reproducible builds. Our pipeline are declared with Python code, the main entrypoint is here. This documentation should be helpful for both local and CI use of the CLI. We indeed power connector testing in the CI with this CLI.

How to install

Requirements

• A running Docker engine with version >= 20.10.23

Install or Update

The recommended way to install airbyte-ci is using the Makefile.

# from the root of the airbyte repository
make tools.airbyte-ci.install

Setting up connector secrets access

If you plan to use Airbyte CI to run CAT (Connector Acceptance Tests), we recommend setting up GSM access so that Airbyte CI can pull remote secrets from GSM. For setup instructions, see the CI Credentials package (which Airbyte CI uses under the hood) README's Get GSM Access instructions.

Updating the airbyte-ci tool

To reinstall airbyte-ci, run the following command:

airbyte-ci update

or if that fails, you can reinstall it with the following command:

# from the root of the airbyte repository
make tools.airbyte-ci.install

Checking the airbyte-ci install

To check that airbyte-ci is installed correctly, run the following command:

make tools.airbyte-ci.check

Cleaning the airbyte-ci install

To clean the airbyte-ci install, run the following command:

make tools.airbyte-ci.clean

Disabling telemetry

We collect anonymous usage data to help improve the tool. If you would like to disable this, you can set the AIRBYTE_CI_DISABLE_TELEMETRY environment variable to true.

Installation for development

Pre-requisites

• Poetry >= 1.1.8
• Python >= 3.10

Installation

If you are developing on pipelines, we recommend installing airbyte-ci with poetry:

cd airbyte-ci/connectors/pipelines/
poetry install
poetry env activate
cd ../../

Alternatively, you can install airbyte-ci with pipx so that the entrypoint is available in your PATH:

make tools.airbyte-ci.install

However, this will not automatically install the dependencies for the local dependencies of airbyte-ci, or respect the lockfile.

Its often best to use the poetry steps instead.

Running Tests

From airbyte-ci/connectors/pipelines:

poetry run pytest tests

You can also run a subset of tests:

poetry run pytest pipelines/models/steps.py

More options, such as running test by keyword matching, are available - see the pytest CLI documentation for all the available options.```

Checking Code Format (Pipelines)

poetry run ruff check pipelines

Commands reference

At this point you can run airbyte-ci commands.

• Airbyte CI CLI
  • What is it?
  • How to install
    • Requirements
  • Install or Update
    • Setting up connector secrets access
    • Updating the airbyte-ci tool
  • Checking the airbyte-ci install
  • Cleaning the airbyte-ci install
  • Disabling telemetry
  • Installation for development
    • Pre-requisites
    • Installation
    • Running Tests
    • Checking Code Format (Pipelines)
  • Commands reference
    • airbyte-ci command group
      • Options
    • connectors command subgroup
      • Options
    • connectors list command
      • Examples
    • connectors test command
      • Examples
      • What it runs
      • Options
      • Extra parameters
    • connectors build command
      • What it runs
    • Options
    • connectors publish command
    • Examples
    • Options
      • What it runs
      • Python registry publishing
    • connectors up-to-date command
    • Examples
    • Other things it could do
    • connectors bump-version command
    • Examples
      • Arguments
    • connectors upgrade_cdk command
    • Examples
      • Arguments
    • connectors migrate-to-base-image command
      • Examples
    • connectors migrate-to-poetry command
      • Examples
    • connectors migrate-to-inline-schemas command
      • Examples
    • connectors pull-request command
      • Examples
    • format command subgroup
    • Options
    • Examples
    • format check all command
    • format fix all command
    • poetry command subgroup
    • Options
    • Examples
    • publish command
      • Options
    • metadata command subgroup
    • metadata deploy orchestrator command
      • Example
      • What it runs
    • tests command
      • Options
      • Examples
    • migrate-to-manifest-only command
      • Examples
  • Changelog
  • More info
• Troubleshooting
  • Commands
    • make tools.airbyte-ci.check
    • make tools.airbyte-ci.clean
  • Common issues
    • airbyte-ci is not found
  • Development
    • airbyte-ci is not found
    • python3.10 not found
    • Any type of pipeline failure

<a id="airbyte-ci-command-group"></a>airbyte-ci command group

The main command group option has sensible defaults. In local use cases you're not likely to pass options to the airbyte-ci command group.

Options

<a id="connectors-command-subgroup"></a>connectors command subgroup

Available commands:

• airbyte-ci connectors test: Run tests for one or multiple connectors.
• airbyte-ci connectors build: Build docker images for one or multiple connectors.
• airbyte-ci connectors publish: Publish a connector to Airbyte's DockerHub.

Options

<a id="connectors-list-command"></a>connectors list command

Retrieve the list of connectors satisfying the provided filters.

Examples

List all connectors:

airbyte-ci connectors list

List all connectors and write the output to a file: airbyte-ci connectors list --output=connectors.json

List certified connectors:

airbyte-ci connectors --support-level=certified list

List connectors changed on the current branch:

airbyte-ci connectors --modified list

List connectors with a specific language:

airbyte-ci connectors --language=python list

List connectors with multiple filters:

airbyte-ci connectors --language=low-code --support-level=certified list

<a id="connectors-test-command"></a>connectors test command

Run a test pipeline for one or multiple connectors.

Examples

Test a single connector: airbyte-ci connectors --name=source-pokeapi test

Test multiple connectors: airbyte-ci connectors --name=source-pokeapi --name=source-bigquery test

Test certified connectors: airbyte-ci connectors --support-level=certified test

Test connectors changed on the current branch: airbyte-ci connectors --modified test

Run acceptance test only on the modified connectors, just run its full refresh tests: airbyte-ci connectors --modified test --only-step="acceptance" --acceptance.-k=test_full_refresh

What it runs

flowchart TD
    entrypoint[[For each selected connector]]
    subgraph static ["Static code analysis"]
      qa[Run QA checks]
      sem["Check version follows semantic versioning"]
      incr["Check version is incremented"]
      metadata_validation["Run metadata validation on metadata.yaml"]
      sem --> incr
    end
    subgraph tests ["Tests"]
        build[Build connector docker image]
        unit[Run unit tests]
        integration[Run integration tests]
        pyairbyte_validation[Python CLI smoke tests via PyAirbyte]
        cat[Run connector acceptance tests]
        secret[Load connector configuration]

        unit-->secret
        unit-->build
        secret-->integration
        secret-->cat
        secret-->pyairbyte_validation
        build-->integration
        build-->cat
    end
    entrypoint-->static
    entrypoint-->tests
    report["Build test report"]
    tests-->report
    static-->report

Options

Note:

• The above options are implemented for Java connectors but may not be available for Python connectors. If an option is not supported, the pipeline will not fail but instead the 'default' behavior will be executed.

Extra parameters

You can pass extra parameters to the following steps:

• unit
• integration
• acceptance

This allows you to override the default parameters of these steps. For example, you can only run the test_read test of the acceptance test suite with: airbyte-ci connectors --name=source-pokeapi test --acceptance.-k=test_read Here the -k parameter is passed to the pytest command running acceptance tests. Please keep in mind that the extra parameters are not validated by the CLI: if you pass an invalid parameter, you'll face a late failure during the pipeline execution.

<a id="connectors-build-command"></a>connectors build command

Run a build pipeline for one or multiple connectors and export the built docker image to the local docker host. It's mainly purposed for local use.

Build a single connector: airbyte-ci connectors --name=source-pokeapi build

Build a single connector with a custom image tag: airbyte-ci connectors --name=source-pokeapi build --tag=my-custom-tag

Build a single connector for multiple architectures: airbyte-ci connectors --name=source-pokeapi build --architecture=linux/amd64 --architecture=linux/arm64

You will get:

• airbyte/source-pokeapi:dev-linux-amd64
• airbyte/source-pokeapi:dev-linux-arm64

Build multiple connectors: airbyte-ci connectors --name=source-pokeapi --name=source-bigquery build

Build certified connectors: airbyte-ci connectors --support-level=certified build

Build connectors changed on the current branch: airbyte-ci connectors --modified build

What it runs

For Python and Low Code connectors:

flowchart TD
    arch(For each platform amd64/arm64)
    connector[Build connector image]
    load[Load to docker host with :dev tag, current platform]
    spec[Get spec]
    arch-->connector-->spec--"if success"-->load

For Java connectors:

flowchart TD
    arch(For each platform amd64/arm64)
    distTar[Gradle distTar task run]
    base[Build integration base]
    java_base[Build integration base Java]
    normalization[Build Normalization]
    connector[Build connector image]

    arch-->base-->java_base-->connector
    distTar-->connector
    normalization--"if supports normalization"-->connector

    load[Load to docker host with :dev tag]
    spec[Get spec]
    connector-->spec--"if success"-->load

Options

<a id="connectors-publish-command"></a>connectors publish command

Run a publish pipeline for one or multiple connectors. It's mainly purposed for CI use to release a connector update.

Examples

Publish all connectors modified in the head commit: airbyte-ci connectors --modified publish

Options

I've added an empty "Default" column, and you can fill in the default values as needed.

What it runs

flowchart TD
    validate[Validate the metadata file]
    check[Check if the connector image already exists]
    build[Build the connector image for all platform variants]
    publish_to_python_registry[Push the connector image to the python registry if enabled]
    upload_spec[Upload connector spec to the spec cache bucket]
    push[Push the connector image from DockerHub, with platform variants]
    pull[Pull the connector image from DockerHub to check SPEC can be run and the image layers are healthy]
    upload_metadata[Upload its metadata file to the metadata service bucket]

    validate-->check-->build-->upload_spec-->publish_to_python_registry-->push-->pull-->upload_metadata

Python registry publishing

If remoteRegistries.pypi.enabled in the connector metadata is set to true, the connector will be published to the python registry. To do so, the --python-registry-token and --python-registry-url options are used to authenticate with the registry and publish the connector. If the current version of the connector is already published to the registry, the publish will be skipped (the --python-registry-check-url is used for the check).

On a pre-release, the connector will be published as a .dev<N> version.

The remoteRegistries.pypi.packageName field holds the name of the used package name. It should be set to airbyte-source-<package name>. Certified Python connectors are required to have PyPI publishing enabled.

An example remoteRegistries entry in a connector metadata.yaml looks like this:

remoteRegistries:
  pypi:
    enabled: true
    packageName: airbyte-source-pokeapi

<a id="connectors-up-to-date"></a>connectors up-to-date command

Meant to be run on a cron script.

Actions:

• Set the latest base image version on selected connectors
• Run poetry update on selected connectors
• Bump the connector version and update the changelog
• Open a PR with the changes, set auto-merge label on it.

Usage: airbyte-ci connectors up-to-date [OPTIONS]

Options:
  --no-bump    Don't bump the version or changelog.
  --dep TEXT  Give a specific set of `poetry add` dependencies to update. For
              example: --dep airbyte-cdk==0.80.0 --dep pytest@^6.2
  --open-reports    Auto open reports in the browser.
  --create-prs      Create pull requests for each updated connector.
  --auto-merge    Set the auto-merge label on created PRs.
  --help      Show this message and exit.

Examples

Get source-openweather up to date. If there are changes, bump the version and add to changelog:

• airbyte-ci connectors --name=source-openweather up-to-date: upgrades main dependecies
• airbyte-ci connectors --name=source-openweather up-to-date
• airbyte-ci connectors --name=source-openweather up-to-date --create-prs: make a pull request for it
• airbyte-ci connectors --name=source-openweather up-to-date --no-bump: don't change the version or changelog

<a id="connectors-bump-version"></a>connectors bump-version command

Bump the version of the selected connectors. A placeholder will be added to the changelog file for the new entry PR number. Use the connectors pull-request command to create a PR, it will update the changelog entry with the PR number.

Examples

Bump source-openweather: airbyte-ci connectors --name=source-openweather bump-version patch "<changelog-entry>"

Arguments

Options

<a id="connectors-upgrade-cdk"></a>connectors upgrade-cdk command

Updates the CDK version of the selected connectors. For Python connectors, sets the airbyte-cdk dependency in pyproject.toml and refreshes the lockfile, updating only essential dependencies.

Examples

airbyte-ci connectors --language=python upgrade-cdk -> Updates all python connectors to the caret range of the latest version. airbyte-ci connectors --name=source-openweather upgrade-cdk "3.0.0" -> Pins source-openweather to version 3.0.0 airbyte-ci connectors --modified upgrade-cdk "<4" -> Updates all modified connectors to the highest available version of major version 3.x.x

Arguments

Notes

When using < (less than) or > (greater than) for the CDK_VERSION argument, it must be wrapped in quotation marks ("<3"). Otherwise the shell (zsh or bash) will interprete these characters as redirection operators.

<a id="connectors-migrate-to-base-image"></a>connectors migrate-to-base-image command

Make a connector using a Dockerfile migrate to the base image by:

• Removing its Dockerfile
• Updating its metadata to use the latest base image version
• Updating its documentation to explain the build process
• Bumping by a patch version

Examples

Migrate source-openweather to use the base image: airbyte-ci connectors --name=source-openweather migrate-to-base-image

<a id="connectors-migrate-to-poetry"></a>connectors migrate-to-poetry command

Migrate connectors the poetry package manager.

Examples

Migrate source-openweather to use the base image: airbyte-ci connectors --name=source-openweather migrate-to-poetry airbyte-ci connectors --name=source-openweather migrate-to-poetry --changelog --bump patch

<a id="connectors-migrate-to-inline-schemas"></a>connectors migrate-to-inline-schemas command

Migrate .json schemas into manifest.yaml files, when present.

Usage: airbyte-ci connectors migrate-to-inline-schemas [OPTIONS]

Options:
  --report  Auto open report browser.
  --help    Show this message and exit.

Examples

Migrate source-quickbooks to use inline schemas: airbyte-ci connectors --name=source-quickbooks migrate-to-inline-schemas

<a id="connectors-pull-request"></a>connectors pull-request command

Makes a pull request for all changed connectors. If the branch already exists, it will update the existing one.

Usage: airbyte-ci connectors pull-request [OPTIONS]

Options:
  -m, --message TEXT          Commit message and pull request title and
                              changelog (if enabled).  [required]
  -b, --branch_id TEXT        update a branch named <branch_id>/<connector-
                              name> instead generating one from the message.
                              [required]
  --report                    Auto open report browser.
  --title TEXT                Title of the PR to be created or edited
                              (optional - defaults to message or no change).
  --body TEXT                 Body of the PR to be created or edited (optional
                              - defaults to empty or not change).
  --help                      Show this message and exit.

Examples

Make a PR for all changes, bump the version and make a changelog in those PRs. They will be on the branch ci_update/round2/<connector-name>: airbyte-ci connectors --modified pull-request -m "upgrading connectors" -b ci_update/round2

Do it just for a few connectors: airbyte-ci connectors --name source-aha --name source-quickbooks pull-request -m "upgrading connectors" -b ci_update/round2

You can also set or set/change the title or body of the PR: airbyte-ci connectors --name source-aha --name source-quickbooks pull-request -m "upgrading connectors" -b ci_update/round2 --title "New title" --body "full body\n\ngoes here"

<a id="connectors-list-command"></a>connectors generate-erd command

Generates a couple of files and publish a new ERD to dbdocs. The generated files are:

• <source code_directory>/erd/discovered_catalog.json: the catalog used to generate the estimated relations and the dbml file
• <source code_directory>/erd/estimated_relationships.json: the output of the LLM trying to figure out the relationships between the different streams
• <source code_directory>/erd/source.dbml: the file used the upload the ERDs to dbdocs

Pre-requisites:

• The config file use to discover the catalog should be available in <source code_directory>/secrets/config.json

Create initial diagram workflow or on connector's schema change

Steps

• Ensure the pre-requisites mentioned above are met
• Run DBDOCS_TOKEN=<token> GENAI_API_KEY=<api key> airbyte-ci connectors --name=<source name> generate-erd
• Create a PR with files <source code_directory>/erd/estimated_relationships.json and <source code_directory>/erd/source.dbml for documentation purposes

Expected Outcome

• The diagram is available in dbdocs
• <source code_directory>/erd/estimated_relationships.json and <source code_directory>/erd/source.dbml are updated on master

On manual validation

Steps

• If not exists, create file <source code_directory>/erd/confirmed_relationships.json with the following format and add:
  • relations describes the relationships that we know exist
  • false_positives describes the relationships the LLM found that we know do not exist

{
    "streams": [
        {
            "name": <stream_name>,
            "relations": {
                <stream_name property>: "<target stream>.<target stream column>"
            }
            "false_positives": {
                <stream_name property>: "<target stream>.<target stream column>"
            }
        },
        <...>
    ]
}

• Ensure the pre-requisites mentioned above are met
• Run DBDOCS_TOKEN=<token> airbyte-ci connectors --name=<source name> generate-erd -x llm_relationships
• Create a PR with files <source code_directory>/erd/confirmed_relationships.json and <source code_directory>/erd/source.dbml for documentation purposes

Options

<a id="format-subgroup"></a>format command subgroup

airbyte-ci format is no longer available. To format code in this repository, we're using pre-commit. Assuming pre-commit is installed, pre-commit run will run the formatters for you.

<a id="poetry-subgroup"></a>poetry command subgroup

Available commands:

• airbyte-ci poetry publish

Options

Examples

• Publish a python package: airbyte-ci poetry --package-path=path/to/package publish --publish-name=my-package --publish-version="1.2.3" --python-registry-token="..." --registry-url="http://host.docker.internal:8012/"

<a id="format-check-command"></a>publish command

This command publishes poetry packages (using pyproject.toml) or python packages (using setup.py) to a python registry.

For poetry packages, the package name and version can be taken from the pyproject.toml file or be specified as options.

Options

<a id="metadata-validate-command-subgroup"></a>metadata command subgroup

Available commands:

• airbyte-ci metadata deploy orchestrator

<a id="metadata-upload-orchestrator"></a>metadata deploy orchestrator command

This command deploys the metadata service orchestrator to production. The DAGSTER_CLOUD_METADATA_API_TOKEN environment variable must be set.

Example

airbyte-ci metadata deploy orchestrator

What it runs

flowchart TD
    test[Run orchestrator tests] --> deploy[Deploy orchestrator to Dagster Cloud]

<a id="tests-command"></a>tests command

This command runs the poe tasks declared in the [tool.airbyte-ci] section of our internal poetry packages. Feel free to checkout this Pydantic model to see the list of available options in [tool.airbyte-ci] section.

You can find the list of internal packages here

Options

Examples

You can pass multiple --poetry-package-path options to run poe tasks.

E.G.: running Poe tasks on the modified internal packages of the current branch: airbyte-ci test --modified

<a id="migrate-to-manifest-only-command"></a>migrate-to-manifest-only command

This command migrates valid connectors to the manifest-only format. It contains two steps:

1. Check: Validates whether a connector is a candidate for the migration. If not, the operation will be skipped.
2. Migrate: Strips out all unneccessary files/folders, leaving only the root-level manifest, metadata, icon, and acceptance/integration test files. Unwraps the manifest (references and $parameters) so it's compatible with Connector Builder.

Examples

airbyte-ci connectors --name=source-pokeapi migrate-to-manifest-only
airbyte-ci connectors --language=low-code migrate-to-manifest-only

Changelog

More info

This project is owned by the Connectors Operations team. We share project updates and remaining stories before its release to production in this EPIC.

Troubleshooting

Commands

make tools.airbyte-ci.check

This command checks if the airbyte-ci command is appropriately installed.

make tools.airbyte-ci.clean

This command removes the airbyte-ci command from your system.

Common issues

airbyte-ci is not found

If you get the following error when running airbyte-ci:

$ airbyte-ci
zsh: command not found: airbyte-ci

It means that the airbyte-ci command is not in your PATH.

Try running

make make tools.airbyte-ci.check

For some hints on how to fix this.

But when in doubt it can be best to run

make tools.airbyte-ci.clean

Then reinstall the CLI with

make tools.airbyte-ci.install

Development

airbyte-ci is not found

To fix this, you can either:

• Ensure that airbyte-ci is installed with pipx. Run pipx list to check if airbyte-ci is installed.
• Run pipx ensurepath to add the pipx binary directory to your PATH.
• Add the pipx binary directory to your PATH manually. The pipx binary directory is usually ~/.local/bin.

python3.10 not found

If you get the following error when running pipx install --editable --force --python=python3.10 airbyte-ci/connectors/pipelines/:

$ pipx install --editable --force --python=python3.10 airbyte-ci/connectors/pipelines/
Error: Python 3.10 not found on your system.

It means that you don't have Python 3.10 installed on your system.

To fix this, you can either:

• Install Python 3.10 with pyenv. Run pyenv install 3.10 to install the latest Python version.
• Install Python 3.10 with your system package manager. For instance, on Ubuntu you can run sudo apt install python3.10.
• Ensure that Python 3.10 is in your PATH. Run which python3.10 to check if Python 3.10 is installed and in your PATH.

Any type of pipeline failure

First you should check that the version of the CLI you are using is the latest one. You can check the version of the CLI with the --version option:

$ airbyte-ci --version
airbyte-ci, version 0.1.0

and compare it with the version in the pyproject.toml file:

$ cat airbyte-ci/connectors/pipelines/pyproject.toml | grep version

If you get any type of pipeline failure, you can run the pipeline with the --show-dagger-logs option to get more information about the failure.

$ airbyte-ci --show-dagger-logs connectors --name=source-pokeapi test

and when in doubt, you can reinstall the CLI with the --force option:

$ pipx reinstall pipelines --force