File syncing

File syncing enables connectors to download and transfer files from API sources when an API endpoint returns a file. This capability supports all common file formats including documents, images, and structured data.

If you're using the Connector Builder UI, file syncing is available from a stream's Advanced > File Uploader option.

Overview

File syncing works in these steps.

Extracting file download URLs from API response records
Making authenticated HTTP requests to download files
Storing files locally with organized naming and metadata
Adding file reference information to each record for destination processing

Schema

yaml

file_uploader:
  title: File Uploader
  description: (experimental) Describes how to fetch a file
  type: object
  required:
    - type
    - requester
    - download_target_extractor
  properties:
    type:
      type: string
      enum: [FileUploader]
    requester:
      description: Requester component that describes how to prepare HTTP requests to send to the source API.
      anyOf:
        - "$ref": "#/definitions/HttpRequester"
        - "$ref": "#/definitions/CustomRequester"
    download_target_extractor:
      description: Responsible for fetching the url where the file is located. This is applied on each records and not on the HTTP response
      anyOf:
        - "$ref": "#/definitions/DpathExtractor"
        - "$ref": "#/definitions/CustomRecordExtractor"
    file_extractor:
      description: Responsible for fetching the content of the file. If not defined, the assumption is that the whole response body is the file content
      anyOf:
        - "$ref": "#/definitions/DpathExtractor"
        - "$ref": "#/definitions/CustomRecordExtractor"
    filename_extractor:
      description: Defines the name to store the file. Stream name is automatically added to the file path. File unique ID can be used to avoid overwriting files. Random UUID will be used if the extractor is not provided.
      type: string
      interpolation_context:
        - config
        - record
      examples:
        - "{{ record.id }}/{{ record.file_name }}/"
        - "{{ record.id }}_{{ record.file_name }}/"
    $parameters:
      type: object
      additionalProperties: true

Required properties

`type`

Set to FileUploader to enable file syncing capability.

`requester`

Defines how to make HTTP requests for downloading files. Supports all standard requester types including authentication methods like OAuth, Bearer tokens, and Basic authentication.

`download_target_extractor`

Extracts the file download URL from each record. Uses DPath syntax to specify the field containing the download URL.

Optional properties

`file_extractor`

Extracts file content from the HTTP response when the entire response body isn't the file. Useful when the API wraps file content in a JSON response.

`filename_extractor`

Customizes file naming using Jinja templating with access to record data and configuration. If not provided, Airbyte uses a random UUID as the filename. It's critical you ensure file names are unique every time.

Authentication

File downloads support independent authentication from the main API requests. This allows downloading files from different domains or services that require separate credentials.

File size limitation

Supports files up to 1.5-GB in size. Airbyte does not sync files larger than that.

Format compatibility

Works with any file format since it handles files as binary data. Common examples:

Documents (PDF, DOCX, TXT)
Images (PNG, JPG, GIF)
Structured data (CSV, JSON, XML)
Archives (ZIP, TAR)

Dynamic URL handling

The url_base can use the special {{download_target}} placeholder to dynamically set the base URL from the extracted download URL, enabling downloads from multiple domains.

Custom file naming

The filename_extractor supports Jinja templating with access to record data, allowing organized file storage with meaningful names and directory structures.

Usage example

Here's how the Zendesk Support connector uses file syncing for article attachments.

yaml

streams:
  article_attachments_stream:
    retriever:
      type: SimpleRetriever
      requester:
        type: HttpRequester
        url_base: "https://{{ config['subdomain'] }}.zendesk.com"
        path: "/api/v2/help_center/articles/{{ stream_partition.parent_id }}/attachments.json"
        http_method: "GET"
        authenticator:
          type: SelectiveAuthenticator
          authenticator_selection_path: ["credentials", "credentials"]
          authenticators:
            oauth2.0: "#/definitions/bearer_authenticator"
            api_token: "#/definitions/basic_authenticator"
      record_selector:
        extractor:
          type: DpathExtractor
          field_path: ["article_attachments"]
    file_uploader:
      type: FileUploader
      requester:
        type: HttpRequester
        url_base: "{{download_target}}"
        http_method: GET
        authenticator:
          type: SelectiveAuthenticator
          authenticator_selection_path: ["credentials", "credentials"]
          authenticators:
            oauth2.0: "#/definitions/bearer_authenticator"
            api_token: "#/definitions/basic_authenticator"
      download_target_extractor:
        type: DpathExtractor
        field_path: ["content_url"]
      filename_extractor: "{{ record.id }}/{{ record.file_name }}/"