ContextQMD
Back to Airbyte

Chroma

docs/integrations/destinations/chroma.md

2.0.010.6 KB
Original Source

Chroma

This page guides you through the process of setting up the Chroma destination connector.

Output Schema

Only one stream will exist to collect data from all source streams. This will be in a collection in Chroma whose name will be defined by the user, and validated and corrected by Airbyte.

For each record, a UUID string is generated and used as the document id. The embeddings generated as defined will be stored as embeddings. Data in the text fields will be stored as documents and those in the metadata fields will be stored as metadata.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedYes
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedYes

Getting Started (Airbyte Open Source)

You can connect to a Chroma instance either in client/server mode or in a local persistent mode. For the local persistent mode, the database file will be saved in the path defined in the path config parameter. Note that path must be an absolute path, prefixed with /local.

:::danger

Persistent Client mode is not supported on Kubernetes

:::

By default, the LOCAL_ROOT env variable in the .env file is set /tmp/airbyte_local.

The local mount is mounted by Docker onto LOCAL_ROOT. This means the /local is substituted by /tmp/airbyte_local by default.

:::caution

Please make sure that Docker Desktop has access to /tmp (and /private on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in Settings -> Resources -> File sharing -> add the one or two above folder and hit the "Apply & restart" button.

:::

Requirements

To use the Chroma destination, you'll need:

  • An account with API access for OpenAI, Cohere (depending on which embedding method you want to use) or neither (if you want to use the default chroma embedding function)
  • A Chroma db instance (client/server mode or persistent mode)
  • Credentials (for cient/server mode)
  • Local File path (for Persistent mode)

Configure Network Access

Make sure your Chroma database can be accessed by Airbyte. If your database is within a VPC, you may need to allow access from the IP you're using to expose Airbyte.

Setup the Chroma Destination in Airbyte

You should now have all the requirements needed to configure Chroma as a destination in the UI. You'll need the following information to configure the Chroma destination:

  • (Required) Text fields to embed
  • (Optional) Text splitter Options around configuring the chunking process provided by the Langchain Python library.
  • (Required) Fields to store as metadata
  • (Required) Collection The name of the collection in Chroma db to store your data
  • (Required) Authentication method
    • For client/server mode
      • Host for example localhost
      • Port for example 8000
      • Username (Optional)
      • Password (Optional)
    • For persistent mode
      • Path The path to the local database file. Note that path must be an absolute path, prefixed with /local.
  • (Optional) Embedding
    • OpenAI API key if using OpenAI for embedding
    • Cohere API key if using Cohere for embedding
    • Embedding Field name and Embedding dimensions if getting the embeddings from stream records

Namespace support

This destination supports namespaces.

Changelog

<details> <summary>Expand to review</summary>
VersionDatePull RequestSubject
0.0.542025-05-0359326Update dependencies
0.0.532025-04-2658256Update dependencies
0.0.522025-04-1257652Update dependencies
0.0.512025-04-0557169Update dependencies
0.0.502025-03-2956624Update dependencies
0.0.492025-03-2256120Update dependencies
0.0.482025-03-0855393Update dependencies
0.0.472025-03-0154852Update dependencies
0.0.462025-02-2254209Update dependencies
0.0.452025-02-1553930Update dependencies
0.0.442025-02-0853428Update dependencies
0.0.432025-02-0152941Update dependencies
0.0.422025-01-2552189Update dependencies
0.0.412025-01-1851744Update dependencies
0.0.402025-01-1151296Update dependencies
0.0.392025-01-0450913Update dependencies
0.0.382024-12-2850445Update dependencies
0.0.372024-12-2150221Update dependencies
0.0.362024-12-1448956Update dependencies
0.0.352024-11-2548668Update dependencies
0.0.342024-11-0448236Update dependencies
0.0.332024-10-2947053Update dependencies
0.0.322024-10-1246434Update dependencies
0.0.312024-09-2846192Update dependencies
0.0.302024-09-2145553Update dependencies
0.0.292024-09-0745322Update dependencies
0.0.282024-08-3145017Update dependencies
0.0.272024-08-2444717Update dependencies
0.0.262024-08-2244530Update test dependencies
0.0.252024-08-1744292Update dependencies
0.0.242024-08-1243734Update dependencies
0.0.232024-08-1043702Update dependencies
0.0.222024-08-0343133Update dependencies
0.0.212024-07-2742628Update dependencies
0.0.202024-07-2042160Update dependencies
0.0.192024-07-1341802Update dependencies
0.0.182024-07-1041384Update dependencies
0.0.172024-07-0941165Update dependencies
0.0.162024-07-0640926Update dependencies
0.0.152024-06-2940634Update dependencies
0.0.142024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.0.132024-06-2540431Update dependencies
0.0.122024-06-2340222Update dependencies
0.0.112024-06-2240068Update dependencies
0.0.102024-04-15#37333Updated CDK & pytest version to fix security vulnerabilities
0.0.92023-12-11#33303Fix bug with embedding special tokens
0.0.82023-12-01#32697Allow omitting raw text
0.0.72023-11-16#32608Support deleting records for CDC sources
0.0.62023-11-13#32357Improve spec schema
0.0.52023-10-23#31563Add field mapping option
0.0.42023-10-15#31329Add OpenAI-compatible embedder option
0.0.32023-10-04#31075Fix OpenAI embedder batch size
0.0.22023-09-29#30820Update CDK
0.0.12023-09-08#30023🎉 New Destination: Chroma (Vector Database)
</details>