Back to Convex Backend

Convex connector for Fivetran

crates/fivetran_source/docs/overview.md

latest5.5 KB
Original Source

Convex {% badge text="Partner-Built" /%} {% availabilityBadge connector="convex" /%}

Convex is a full-stack TypeScript development platform. Replace your database, server functions, and glue code.

{% note %} This connector is partner-built. For any questions related to the Convex connector and its documentation, refer to Convex's support team. For details on SLA, see Convex's Status and Guarantees documentation. {% /note %}


Features

{% featureTable connector="convex" /%}


{% partial file="saas-supported-deployment-models.template.md" /%}


Setup guide

Follow the step-by-step Convex setup guide to connect your Convex database with Fivetran.


Sync overview

Once Fivetran is connected to your Convex deployment, the connection fetches an initial consistent snapshot of all data from your Convex database. Once the initial sync is complete, the connection uses Change data capture (CDC) to efficiently incrementally sync updates at a newer consistent view of your Convex deployment. You can configure the frequency of these updates.


Configuration

To configure a Convex connection, you need your deployment URL and deploy key. You can find both on your project's Production Deployment Settings page.


Schema information

Fivetran tries to replicate the database and columns from your configured Convex deployment to your destination according to Fivetran's standard database update strategies.

Type transformations and mapping

As the connection extracts your data, it matches Convex data types to types that Fivetran supports.

The following table illustrates how the connection transforms your Convex data types into Fivetran-supported types:

Convex TypeFivetran TypeFivetran Supported
IdSTRINGTrue
NullNULLTrue
Int64LONGTrue
Float64DOUBLETrue
BooleanBOOLEANTrue
StringSTRINGTrue
BytesBINARYTrue
ArrayJSONTrue
ObjectJSONTrue

{% note %} The _creationTime system field in each document is special-cased to convert into a UTC_DATETIME, despite being stored as a Float64 inside of Convex. {% /note %}

{% note %} Nested types inside Object and Array are serialized as JSON using the JSON format for export. {% /note %}

Nested data

Convex documents are represented as JSON by using conversions. If the first-level field is a simple data type, the connection will map it to its own type. If it's a complex nested data type such as an array or JSON data, it maps to a JSON type without unpacking. The connection does not automatically unpack nested JSON objects to separate tables in the destination. Any nested JSON objects are preserved as is in the destination so that you can use JSON processing functions.

For example, the following Convex document:

json
{"street"  : "Main St."
"city"     : "New York"
"country"  : "US"
"phone"    : "(555) 123-5555"
"zip code" : 12345
"people"   : ["John", "Jane", "Adam"]
"car"      : {"make" : "Honda",
              "year" : 2014,
              "type" : "AWD"}
}

is converted to the following table when the connection loads it into your destination:

_idstreetcitycountryphonezip codepeoplecar
1Main St.New YorkUS(555) 123-555512345["John", "Jane", "Adam"]{"make" : "Honda", "year" : 2014, "type" : "AWD"}

Fivetran-generated data

Fivetran adds the following column to every table in your destination:

  • _fivetran_synced (UTC TIMESTAMP) indicates the time when Fivetran last successfully synced the row. It is added to every table.
  • _fivetran_deleted (BOOLEAN) indicates if the column was deleted in the source.

Fivetran adds these columns to give you insight into the state of your data and the progress of your data syncs. For more information about these columns, see our System Columns and Tables documentation.

Excluding source data

If you don’t want to sync all the data from your Convex database, you can exclude Convex components, tables, and fields from your syncs on your Fivetran dashboard. To do so, go to your Convex connection details page and uncheck the objects you would like to omit from syncing.

For more information, see our Data Blocking documentation.

{% important %} If you modify your source data inclusion settings after data has already been synced, the existing data will not be modified retroactively. If you need to apply the new settings to the existing data, trigger a historical re-sync. {% /important %}