SMMF (Service-oriented Multi-Model Management Framework)

SMMF is DB-GPT's model management layer. It provides a unified interface for managing, switching, and deploying multiple LLM and embedding models — whether they are API proxies or locally hosted.

Why SMMF?

Different tasks benefit from different models. SMMF lets you:

• Run multiple models simultaneously (e.g., one for chat, one for embeddings)
• Switch models without code changes — just update config
• Scale independently — deploy models on separate machines in cluster mode
• Mix providers — use OpenAI for chat and a local model for embeddings

Supported providers

API Proxy

Local Inference

Configuration

Models are configured in TOML files under configs/:

[models]

# LLM configuration
[[models.llms]]
name = "chatgpt_proxyllm"
provider = "proxy/openai"
api_key = "sk-..."

# Embedding model configuration
[[models.embeddings]]
name = "text-embedding-3-small"
provider = "proxy/openai"
api_key = "sk-..."

You can define multiple LLMs and embeddings in the same config file.

Deployment modes

Standalone

All models run in the same process as the DB-GPT server. Simple and suitable for development or single-machine deployments.

uv run dbgpt start webserver --config configs/dbgpt-proxy-openai.toml

Cluster

Models run on separate worker nodes, managed by a controller. Suitable for production deployments with multiple GPUs or machines.

flowchart LR
    Server["DB-GPT Server"] --> Controller["Model Controller"]
    Controller --> Worker1["Worker (LLM)"]
    Controller --> Worker2["Worker (Embedding)"]
    Controller --> Worker3["Worker (LLM 2)"]

Learn more: Cluster Deployment

What's next

• Model Providers — Detailed setup for each provider
• SMMF Module — Deep dive into multi-model management
• Cluster Deployment — Scale with multiple workers