How to generate embeddings

This page shows how to:

• Generate embeddings with a unified API. TensorZero unifies many LLM APIs (e.g. OpenAI) and inference servers (e.g. Ollama).
• Use any programming language. You can use any OpenAI SDK (Python, Node, Go, etc.) or the OpenAI-compatible HTTP API.

You can find a complete runnable example of this guide on GitHub.

Generate embeddings from OpenAI

Our example uses the OpenAI Python SDK, but you can use any OpenAI SDK or call the OpenAI-compatible HTTP API. See Call any LLM for an example using the OpenAI Node SDK.

The TensorZero Python SDK doesn't have an independent embedding endpoint at the moment.

You can point the OpenAI Python SDK to a TensorZero Gateway to generate embeddings with a unified API.

For example, if you're using OpenAI, you can set the OPENAI_API_KEY environment variable with your API key.

export OPENAI_API_KEY="sk-..."

See the Integrations page to learn how to set up credentials for other LLM providers.

You can install the OpenAI SDK with a Python package manager like pip.

pip install openai

Let's deploy the TensorZero Gateway using Docker. For simplicity, we'll use the gateway without observability or custom configuration.

docker run \
  -e OPENAI_API_KEY \
  -p 3000:3000 \
  tensorzero/gateway \
  --default-config

See the Deploy the TensorZero Gateway page for more details.

Let's initialize the OpenAI SDK and point it to the gateway we just launched.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:3000/openai/v1", api_key="not-used")

result = client.embeddings.create(
    input="Hello, world!",
    model="tensorzero::embedding_model_name::openai::text-embedding-3-small",
    # or: Azure, any OpenAI-compatible endpoint (e.g. Ollama, Voyager)
)

CreateEmbeddingResponse(
    data=[
        Embedding(
            embedding=[
                -0.019143931567668915,
                # ...
            ],
            index=0,
            object='embedding'
        )
    ],
    model='tensorzero::embedding_model_name::openai::text-embedding-3-small',
    object='list',
    usage=Usage(prompt_tokens=4, total_tokens=4)
)

Define a custom embedding model

You can define a custom embedding model in your TensorZero configuration file.

For example, let's define a custom embedding model for nomic-embed-text served locally by Ollama.

Download the embedding model and launch the Ollama server:

ollama pull nomic-embed-text
ollama serve

We assume that Ollama is running on your host machine at http://localhost:11434.

Add your custom model and model provider to your configuration file:

[embedding_models.nomic-embed-text]
routing = ["ollama"]

[embedding_models.nomic-embed-text.providers.ollama]
type = "openai"
api_base = "http://host.docker.internal:11434/v1"
model_name = "nomic-embed-text"
api_key_location = "none"

See the Configuration Reference for details on configuring your embedding models.

Deploy the TensorZero Gateway with your configuration file. Make sure that the container has access to the Ollama server running on the host.

See the Deploy the TensorZero Gateway page for more details.

Use your custom model by referencing it with tensorzero::embedding_model_name::nomic-embed-text.

For example, using the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:3000/openai/v1", api_key="not-used")

result = client.embeddings.create(
    input="Hello, world!",
    model="tensorzero::embedding_model_name::nomic-embed-text",
)

CreateEmbeddingResponse(
    data=[
        Embedding(
            embedding=[
                -0.019143931567668915,
                # ...
            ],
            index=0,
            object='embedding'
        )
    ],
    model='tensorzero::embedding_model_name::nomic-embed-text',
    object='list',
    usage=Usage(prompt_tokens=4, total_tokens=4)
)

Cache embeddings

The TensorZero Gateway supports caching embeddings to improve latency and reduce costs. When caching is enabled, identical embedding requests will be served from the cache instead of being sent to the model provider.

result = client.embeddings.create(
    input="Hello, world!",
    model="tensorzero::embedding_model_name::openai::text-embedding-3-small",
    extra_body={
        "tensorzero::cache_options": {
            "enabled": "on",  # Enable reading from and writing to cache
            "max_age_s": 3600,  # Optional: cache entries older than 1 hour are ignored
        }
    }
)

Caching works for single embeddings. Batch embedding requests (multiple inputs) will write to the cache but won't serve cached responses.

See the Inference Caching guide for more details on cache modes and options.