Conversational search

Conversational search lets Manticore Buddy answer questions over an existing vectorized table. Buddy retrieves the most relevant rows with KNN search, turns those rows into context, and sends the context plus the conversation history to an LLM.

It is managed from SQL with:

• CREATE CHAT MODEL
• SHOW CHAT MODELS
• DESCRIBE CHAT MODEL
• DROP CHAT MODEL
• CALL CHAT

Before you start

You need a vectorized table and an LLM provider. The table requirements are covered below. Provider credentials can be set in CREATE CHAT MODEL with api_key, or supplied through the matching environment variable, such as OPENAI_API_KEY.

How it works

When CALL CHAT runs, Buddy builds a retrieval-augmented answer in this order:

1. Buddy loads the chat model.
2. Buddy loads the conversation history for the supplied conversation UUID. If no UUID is supplied, it creates one.
3. Buddy inspects the target table and chooses a FLOAT_VECTOR field.
4. The LLM decides how to handle the message: search again, answer from the previous search context, or answer without retrieval.
5. Buddy runs KNN search with the selected vector field when retrieval is needed.
6. Buddy builds the LLM context from the vector field's from='...' source fields.
7. The configured LLM generates the answer.
8. Buddy saves the user message and the assistant reply in the conversation history.

The fifth argument of CALL CHAT is called fields internally, but for conversational search it means the vector field used by knn(...). It is not a list of fields to return. Buddy selects rows with SELECT *, then removes vector columns from the sources payload so the response does not include large embedding values.

Table requirements

The table must have at least one FLOAT_VECTOR field configured for auto embeddings. The vector field must include from='...', because Buddy uses those source fields as LLM context.

The examples below use onnx-models/all-MiniLM-L12-v2-onnx, which runs through the recommended ONNX path and does not require an embedding API key.

SQL:

CREATE TABLE docs (
    id BIGINT,
    title TEXT,
    content TEXT,
    embedding FLOAT_VECTOR
        knn_type='hnsw'
        hnsw_similarity='cosine'
        model_name='onnx-models/all-MiniLM-L12-v2-onnx'
        from='title,content'
) TYPE='rt';

INSERT INTO docs(id, title, content) VALUES
    (1, 'Vector search', 'Vector search compares embeddings to find semantically similar documents.'),
    (2, 'Full-text search', 'Full-text search matches terms and phrases in indexed text.');

If CALL CHAT does not specify a vector field, Buddy uses the first FLOAT_VECTOR field found in the table definition.

Creating a chat model

Use CREATE CHAT MODEL to store the LLM provider, model id, and retrieval settings.

SQL:

CREATE CHAT MODEL assistant (
    model='openai:gpt-4o-mini'
);

You can also set provider options and retrieval limits:

SQL:

CREATE CHAT MODEL support_assistant (
    model='openai:gpt-4o-mini',
    api_key='your-provider-api-key',
    base_url='http://host.docker.internal:8787/v1',
    timeout=60,
    retrieval_limit=5,
    max_document_length=3000
);

Common options:

Chat model names may contain only letters, numbers, and underscores.

The model option must use provider:model format:

model='openai:gpt-4o-mini'

Provider api_key is optional if the provider key is already available in Buddy's environment. For example, a Docker Compose service can pass provider keys like this:

environment:
  - OPENAI_API_KEY=${OPENAI_API_KEY}
  - OPENROUTER_API_KEY=${OPENROUTER_API_KEY}

If api_key is not set in CREATE CHAT MODEL, the llm extension can use the matching provider environment variable. Set api_key in the chat model only when you need this model to use a different key.

CALL CHAT syntax

CALL CHAT(
    'query',
    'table',
    'model_name',
    'conversation_uuid',
    'vector_field'
);

Arguments are positional only:

The table argument must be a plain table identifier, optionally qualified as database.table. The vector field argument must be a plain field identifier.

Asking questions

Use CALL CHAT with a query, a table, and a chat model.

SQL:

CALL CHAT(
    'What is vector search?',
    'docs',
    'assistant'
);

To continue a conversation, pass the same conversation UUID:

SQL:

CALL CHAT(
    'Can you explain it with an example?',
    'docs',
    'assistant',
    'docs-chat-001'
);

To search a specific vector field, pass it as the fifth argument:

SQL:

CALL CHAT(
    'Find documents where the title is about vector search',
    'docs',
    'assistant',
    '',
    'title_embedding'
);

When the fifth argument is present, Buddy checks that the field exists and is a FLOAT_VECTOR. If the argument is omitted, Buddy detects the first FLOAT_VECTOR field from SHOW CREATE TABLE.

Search and context details

When Buddy needs retrieval, it runs KNN search on the selected vector field and returns up to retrieval_limit rows. The default distance threshold is 0.8.

Buddy uses the retrieved rows as LLM context. The same rows are returned in sources, with knn_dist included and FLOAT_VECTOR columns removed.

max_document_length limits how much text from each source row can be sent to the LLM. Use 0 to disable truncation; otherwise use a value from 100 to 65536.

Response

CALL CHAT returns one row:

Example response shape:

{
  "conversation_uuid": "docs-chat-001",
  "user_query": "What is vector search?",
  "search_query": "vector search, embeddings, similarity search",
  "response": "Vector search finds similar items by comparing embeddings...",
  "sources": "[{\"id\":1,\"title\":\"Vector Search\",\"content\":\"...\",\"knn_dist\":0.12}]"
}

Vector fields are not included in sources.

Managing chat models

List models:

SQL:

SHOW CHAT MODELS;

Describe a model:

SQL:

DESCRIBE CHAT MODEL assistant;

Drop a model:

SQL:

DROP CHAT MODEL assistant;

Drop safely:

SQL:

DROP CHAT MODEL IF EXISTS assistant;

SHOW CHAT MODELS returns name, model, and created_at. DESCRIBE CHAT MODEL returns property and value; stored API keys are shown as HIDDEN.

Dropping a chat model also drops that model's conversation history table. Conversation history is stored per model and written with a 30-day TTL.