Azure Blob Storage

Overview

This destination writes data to Azure Blob Storage.

The Airbyte Azure Blob Storage destination allows you to sync data to Azure Blob Storage. Each stream is written to its own blob under the container, as <stream_namespace>/<stream_name>/yyyy_mm_dd_<unix_epoch>_<part_number>.<file_extension>.

Supported sync modes

Configuration

Output Schema

CSV

Like most other Airbyte destination connectors, the output contains your data, along with some metadata fields. If you select the "root level flattening" option, your data will be promoted to additional columns; if you select "no flattening", your data will be left as a JSON blob inside the _airbyte_data column.

For example, given the following JSON object from a source:

{
  "user_id": 123,
  "name": {
    "first": "John",
    "last": "Doe"
  }
}

With no flattening, the output CSV is:

With root level flattening, the output CSV is:

JSON Lines (JSONL)

JSON Lines is a text format with one JSON per line. As with the CSV format, this connector will write your data along with some metadata fields. You can enable "root level flattening" to promote your data to the root of the JSON object, or use "no flattening" to leave your data inside the _airbyte_data object.

For example, given the following two JSON object from a source:

{
  "user_id": 123,
  "name": {
    "first": "John",
    "last": "Doe"
  }
}
{
  "user_id": 456,
  "name": {
    "first": "Jane",
    "last": "Roe"
  }
}

With no flattening, the output JSONL is:

{ "_airbyte_raw_id": "26d73cde-7eb1-4e1e-b7db-a4c03b4cf206", "_airbyte_extracted_at": "1622135805000", "_airbyte_generation_id": "11", "_airbyte_meta": { "changes": [], "sync_id": 10111 }, "_airbyte_data": { "user_id": 123, "name": { "first": "John", "last": "Doe" } } }
{ "_airbyte_raw_id": "0a61de1b-9cdd-4455-a739-93572c9a5f20", "_airbyte_extracted_at": "1631948170000", "_airbyte_generation_id": "12", "_airbyte_meta": { "changes": [], "sync_id": 10112 }, "_airbyte_data": { "user_id": 456, "name": { "first": "Jane", "last": "Roe" } } }

With root level flattening, the output JSONL is:

{ "_airbyte_raw_id": "26d73cde-7eb1-4e1e-b7db-a4c03b4cf206", "_airbyte_extracted_at": "1622135805000", "_airbyte_generation_id": "11", "_airbyte_meta": { "changes": [], "sync_id": 10111 }, "user_id": 123, "name": { "first": "John", "last": "Doe" } }
{ "_airbyte_raw_id": "0a61de1b-9cdd-4455-a739-93572c9a5f20", "_airbyte_extracted_at": "1631948170000", "_airbyte_generation_id": "12", "_airbyte_meta": { "changes": [], "sync_id": 10112 }, "user_id": 456, "name": { "first": "Jane", "last": "Roe" } }

Getting started

Requirements

1. Create an AzureBlobStorage account.
2. Check if it works under https://portal.azure.com/ -> "Storage browser".

Setup guide

• Fill up AzureBlobStorage info
  • Azure Blob Storage Endpoint Domain Name
    • Leave the default value (or leave it empty if running the container from the command line) to use the Microsoft endpoint, or specify your own.
  • Azure Blob Storage Container Name
    • If the container does not exist, it will be created automatically. If left empty, a container named airbytecontainer with a timestamp suffix will be created.
  • Azure Blob Storage Account Name
    • See this on how to create an account.
  • Authentication - you must use exactly one of these:
    • Shared Access Signature (recommended)
      • See this for how to create an SAS.
    • Azure Entra ID (Service Principal)
      • Azure Tenant ID, Azure Client ID, and Azure Client Secret from an Azure service principal with appropriate permissions.
      • See this for how to create a service principal.
    • Azure Blob Storage Account Key
      • Corresponding key to the above user.
  • Format
    • Data format that will be use for a migrated data representation in blob.
• Make sure your user has access to Azure from the machine running Airbyte.
  • This depends on your networking setup.
  • The easiest way to verify if Airbyte is able to connect to your Azure blob storage container is via the check connection tool in the UI.

Namespace support

This destination supports namespaces. The namespace is used as part of the output path structure.

Changelog