ContextQMD
Back to Airbyte

Firebolt

docs/integrations/destinations/firebolt.md

2.0.09.9 KB
Original Source

Firebolt

This page guides you through the process of setting up the Firebolt destination connector.

Prerequisites

  1. Firebolt Account: You need an active Firebolt account. Sign up here if you don’t have one.
  2. Firebolt Database and Engine created: Follow the guide to create an engine if you don't have one in your Firebolt workspace.
  3. Service Account credentials: all programmatic access to Firebolt requires a service account to interact with the database. Create a service account as described in this guide and note its ID and secret.

Replication strategies

This Firebolt destination connector has two replication strategies:

  1. SQL: Replicates data via SQL INSERT queries. This leverages Firebolt SDK to execute queries directly on Firebolt Engines. Not recommended for production workloads as this does not scale well.

  2. S3: Replicates data by first uploading data to an S3 bucket, creating an External Table and writing into a final Fact Table. This is the recommended loading approach. Requires an S3 bucket and credentials in addition to Firebolt credentials.

Connection parameters

  • Client ID
    • Your service account id.
  • Client Secret
    • Your service account secret.
  • Account
  • Database
    • The database name where you want the data to be stored.
  • Engine
    • Name of the engine that will run the queries.
  • Host (optional)
    • Firebolt backend URL. Can be left blank for most usecases.

Additionally, for S3 strategy:

  • S3 Bucket Name
    • See this to create an S3 bucket.
  • S3 Bucket Region
    • Create the S3 bucket on the same region as the Firebolt database.
  • Access Key Id
  • Secret Access Key
    • Corresponding key to the above key id.

Setup guide

  1. Sign up to Firebolt following the guide
  2. Follow the getting started tutorial to setup a database.
  3. Create a service account.
  4. Create an engine as described in here
  5. (Optional) Create a staging S3 bucket (for the S3 strategy).
  6. (Optional) Create an IAM with programmatic access to read, write and delete objects from an S3 bucket.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Connector-specific features & highlights

Output schema

Each stream will be output into its own raw Fact table in Firebolt. Each table will contain 3 columns:

  • _airbyte_ab_id: a uuid assigned by Airbyte to each event that is processed. The column type in Firebolt is VARCHAR.
  • _airbyte_emitted_at: a timestamp representing when the event was pulled from the data source. The column type in Firebolt is TIMESTAMP.
  • _airbyte_data: a json blob representing the event data. The column type in Firebolt is VARCHAR but can be parsed with JSON functions.

Namespace support

This destination does not support namespaces.

Changelog

<details> <summary>Expand to review</summary>
VersionDatePull RequestSubject
0.2.402025-05-2459866Update dependencies
0.2.392025-05-0359316Update dependencies
0.2.382025-04-2658725Update dependencies
0.2.372025-04-1957623Update dependencies
0.2.362025-04-0557115Update dependencies
0.2.352025-03-2956607Update dependencies
0.2.342025-03-2256156Update dependencies
0.2.332025-03-0855391Update dependencies
0.2.322025-03-0154853Update dependencies
0.2.312025-02-2254217Update dependencies
0.2.302025-02-1553940Update dependencies
0.2.292025-02-0853427Update dependencies
0.2.282025-02-0152946Update dependencies
0.2.272025-01-2549292Update dependencies
0.2.262025-01-1751560Fix connection issues
0.2.252024-11-2548672Update dependencies
0.2.242024-10-2947780Update dependencies
0.2.232024-10-2847100Update dependencies
0.2.222024-10-1246841Update dependencies
0.2.212024-10-0546420Update dependencies
0.2.202024-09-2846144Update dependencies
0.2.192024-09-2145744Update dependencies
0.2.182024-09-1445562Update dependencies
0.2.172024-09-0745245Update dependencies
0.2.162024-08-3144991Update dependencies
0.2.152024-08-2444698Update dependencies
0.2.142024-08-2244530Update test dependencies
0.2.132024-08-1744239Update dependencies
0.2.122024-08-1043682Update dependencies
0.2.112024-08-0343143Update dependencies
0.2.102024-07-2742703Update dependencies
0.2.92024-07-2042211Update dependencies
0.2.82024-07-1341789Update dependencies
0.2.72024-07-1041602Update dependencies
0.2.62024-07-0941118Update dependencies
0.2.52024-07-0640854Update dependencies
0.2.42024-06-2740578Replaced deprecated AirbyteLogger with logging.Logger
0.2.32024-06-2540494Update dependencies
0.2.22024-06-2240078Update dependencies
0.2.12024-06-0639157[autopull] Upgrade base image to v1.2.2
0.2.02024-05-0836443Service account authentication support
0.1.12024-03-0535838Un-archive connector
0.1.02022-05-1813118New Destination: Firebolt
</details>