ContextQMD
Back to Airbyte

Senseforce

docs/integrations/sources/senseforce.md

2.0.010.1 KB
Original Source

Senseforce

This page guides you through the process of setting up the Senseforce source connector.

Sync overview

Prerequisites

Creating a Senseforce Dataset to Export

The Senseforce Airbyte connector allows to export custom datasets built bei Senseforce users. Follow these steps to configure a dataset which can be exported with the Airbyte connector:

  1. Create a new, empty dataset as documented here
  2. Add at least the following columns (these columns are Senseforce system columns and available for all of your custom data models/event schemas):
    1. Metadata -> Timestamp
    2. Metadata -> Thing
    3. Metadata -> Id
  3. Add any other column of your event schemas you want to export
  4. Enter a descriptive Name and a Description and save the dataset
  5. Note the ID of the dataset (the GUID at the end of the URL path of your dataset in your browser URL bar)

Tip: For most exports it is recommended to have the Timestamp column in first place. The Airbyte connector automatically orders in ascending direction. If the Timestamp column is not in the first position, incremental syncs might not work properly.

IMPORTANT: The Timestamp, Thing and Id column are mandatory for the Connector to work as intended. While it still works without eg. the "Id", functionality might be impaired if one of these 3 columns is missing. Make sure to not rename these columns - keep them at their default names.

Set up the Senseforce source connector

  1. Log into your Airbyte Cloud or Airbyte Open Source account.

  2. Click Sources and then click + New source.

  3. On the Set up the source page, select Senseforce from the Source type dropdown.

  4. Enter a name for your source.

  5. For API Access Token, enter your Senseforce API Access Token.

  6. For Senseforce backend URL, enter your Senseforce Backend URL.

  7. For Dataset ID, enter your Senseforce Dataset ID.

    We recommend creating an api access token specifically for Airbyte to control which resources Airbyte can access. For good operations, we recommend to create a separate Airbyte User as well as a separate Senseforce Airbyte Group. Share the dataset with this group and grant Dataset Read, Event Schema Read and Machine Master Data Read permissions.

  8. For The first day (in UTC) when to read data from, enter the day in YYYY-MM-DD format. The data added on and after this day will be replicated.

  9. Click Set up source.

Supported sync modes

The Senseforce source connector supports the following sync modes:

  • Full Refresh
  • Incremental

NOTE: The Senseforce Airbyte connector uses the Timestamp column to determine, which data were already read. Data inserted AFTER a finished sync, with timestamps less than already synced ones, are not considered for the next sync anymore. If this behavior does not fit your use case, follow the next section

Using Inserted Timestamp instead of Data Timestamp for incremental modes

  1. Rename your "Timestamp" column to "Timestamp_data"
  2. Add the Metadata -> Inserted column to your dataset.
  3. Move the newly added "Inserted" column to position 1.
  4. Rename the "Inserted" column to "Timestamp".

Now the inserted timestamp will be used for creating the Airbyte cursor. Note that this method results in slower syncs, as the Senseforce queries to generate the Datasets are slower than if you use the data timestamp.

Supported Streams

The Senseforce source connector supports the following streams:

Features

FeatureSupported?(Yes/No)Notes
Full Refresh SyncYes
Incremental SyncYes

Performance considerations

Senseforce utilizes an undocumented rate limit which - under normal use - should not be triggered, even with huge datasets. Create an issue if you see any rate limit issues that are not automatically retried successfully.

Changelog

<details> <summary>Expand to review</summary>
VersionDatePull RequestSubject
0.2.272025-05-2460492Update dependencies
0.2.262025-05-1060180Update dependencies
0.2.252025-05-0459594Update dependencies
0.2.242025-04-2759027Update dependencies
0.2.232025-04-1958378Update dependencies
0.2.222025-04-1258005Update dependencies
0.2.212025-04-0557476Update dependencies
0.2.202025-03-2956906Update dependencies
0.2.192025-03-2256323Update dependencies
0.2.182025-03-0855611Update dependencies
0.2.172025-03-0155006Update dependencies
0.2.162025-02-2354562Update dependencies
0.2.152025-02-1554005Update dependencies
0.2.142025-02-0853498Update dependencies
0.2.132025-02-0152957Update dependencies
0.2.122025-01-2552483Update dependencies
0.2.112025-01-1851912Update dependencies
0.2.102025-01-1151356Update dependencies
0.2.92024-12-2850738Update dependencies
0.2.82024-12-2150235Update dependencies
0.2.72024-12-1449662Update dependencies
0.2.62024-12-1249341Update dependencies
0.2.52024-12-1149100Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64
0.2.42024-11-0448166Update dependencies
0.2.32024-10-2947765Update dependencies
0.2.22024-10-2847567Update dependencies
0.2.12024-08-1644196Bump source-declarative-manifest version
0.2.02024-08-1444073Refactor connector to manifest-only format
0.1.152024-08-1243865Update dependencies
0.1.142024-08-1043704Update dependencies
0.1.132024-08-0343274Update dependencies
0.1.122024-07-2742763Update dependencies
0.1.112024-07-2042247Update dependencies
0.1.102024-07-1341792Update dependencies
0.1.92024-07-1041402Update dependencies
0.1.82024-07-0941242Update dependencies
0.1.72024-07-0640914Update dependencies
0.1.62024-06-2540380Update dependencies
0.1.52024-06-2240190Update dependencies
0.1.42024-06-1539462Make compatible with builder
0.1.32024-06-0439073[autopull] Upgrade base image to v1.2.1
0.1.22024-05-2038431[autopull] base image + poetry + up_to_date
0.1.12023-02-1322892Specified date formatting in specification
0.1.02022-10-26#18775🎉 New Source: Mailjet SMS API [low-code CDK]
</details>