Chroma BM25

import { Callout } from '/snippets/callout.mdx';

Chroma provides a built-in BM25 sparse embedding function. BM25 (Best Matching 25) is a ranking function used to estimate the relevance of documents to a given search query. This embedding function runs locally and does not require any external API keys.

Sparse embeddings are useful for retrieval tasks where you want to match on specific keywords or terms, rather than semantic similarity.

This embedding function uses snowballstemmer to tokenize documents.

pip install snowballstemmer

from chromadb.utils.embedding_functions import ChromaBm25EmbeddingFunction

bm25_ef = ChromaBm25EmbeddingFunction(
    k=1.2,
    b=0.75,
    avg_doc_length=256.0,
    token_max_length=40
)

texts = ["Hello, world!", "How are you?"]
sparse_embeddings = bm25_ef(texts)

You can customize the BM25 parameters:

• k: Controls term frequency saturation (default: 1.2)
• b: Controls document length normalization (default: 0.75)
• avg_doc_length: Average document length in tokens (default: 256.0)
• token_max_length: Maximum token length (default: 40)
• stopwords: Optional list of stopwords to exclude

// npm install @chroma-core/chroma-bm25

import { ChromaBm25EmbeddingFunction } from "@chroma-core/chroma-bm25";

const embedder = new ChromaBm25EmbeddingFunction({
  k: 1.2,
  b: 0.75,
  avgDocLength: 256.0,
  tokenMaxLength: 40,
});

// use directly
const sparseEmbeddings = await embedder.generate(["document1", "document2"]);

You can customize the BM25 parameters:

• k: Controls term frequency saturation (default: 1.2)
• b: Controls document length normalization (default: 0.75)
• avgDocLength: Average document length in tokens (default: 256.0)
• tokenMaxLength: Maximum token length (default: 40)
• stopwords: Optional list of stopwords to exclude

use chroma::embed::bm25::BM25SparseEmbeddingFunction;

let bm25 = BM25SparseEmbeddingFunction::default_murmur3_abs();
let sparse_vector = bm25.encode("document text")?;