docs/serving/online_serving/README.md
vLLM provides an HTTP server that is compatible with many interfaces!
We currently support the following OpenAI APIs:
/v1/completions)
suffix parameter is not supported./v1/responses)
/v1/chat/completions)
user parameter is ignored.parallel_tool_calls parameter to false ensures vLLM only returns zero or one tool call per request. Setting it to true (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to true, as that behavior is model dependent and not all models are designed to support parallel tool calls./v1/embeddings)
/v1/audio/transcriptions)
/v1/audio/translations)
/v1/messages)/v2/embed)
/rerank, /v1/rerank, /v2/rerank)
/invocations - SageMaker-compatible endpoint (routes to the same inference functions as /v1 endpoints)For further details on pooling models, please refer to this page.
/classify)/v2/embed)/v1/embeddings)/score)/rerank, /v1/rerank, /v2/rerank)/pooling)
For further details on speech to text, please refer to this page.
/v1/audio/transcriptions)
/v1/audio/translations)
/v1/realtime)
For further details on renderer APIs, please refer to this page.
/v1/completions/render)
/v1/chat/completions/render)
/classify)
/score, /v1/score)
/pooling)
/generative_scoring)
"generate").label_token_ids./tokenize - Tokenize text/detokenize - Detokenize tokens/health - Health check/ping - SageMaker health check/version - Version information/load - Server load metricsFor further details on sleep mode, please refer to this page.
/sleep - Put engine to sleep (causes denial of service)/wake_up - Wake engine from sleep/is_sleeping - Check if engine is sleeping/collective_rpc - Execute arbitrary RPC methods on the engine (extremely dangerous)In order for the language model to support chat protocol, vLLM requires the model to include a chat template in its tokenizer configuration. The chat template is a Jinja2 template that specifies how roles, messages, and other chat-specific tokens are encoded in the input.
An example chat template for NousResearch/Meta-Llama-3-8B-Instruct can be found here
Some models do not provide a chat template even though they are instruction/chat fine-tuned. For those models,
you can manually specify their chat template in the --chat-template parameter with the file path to the chat
template, or the template in string form. Without a chat template, the server will not be able to process chat
and all chat requests will error.
vllm serve <model> --chat-template ./path-to-chat-template.jinja
vLLM community provides a set of chat templates for popular models. You can find them under the examples directory.
With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies
both a type and a text field. An example is provided below:
completion = client.chat.completions.create(
model="NousResearch/Meta-Llama-3-8B-Instruct",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Classify this sentiment: vLLM is wonderful!"},
],
},
],
)
Most chat templates for LLMs expect the content field to be a string, but there are some newer models like
meta-llama/Llama-Guard-3-1B that expect the content to be formatted according to the OpenAI schema in the
request. vLLM provides best-effort support to detect this automatically, which is logged as a string like
"Detected the chat template content format to be...", and internally converts incoming requests to match
the detected format, which can be one of:
"string": A string.
"Hello world""openai": A list of dictionaries, similar to OpenAI schema.
[{"type": "text", "text": "Hello world!"}]If the result is not what you expect, you can set the --chat-template-content-format CLI argument
to override which format to use.
The FastAPI /docs endpoint requires an internet connection by default. To enable offline access in air-gapped environments, use the --enable-offline-docs flag:
vllm serve NousResearch/Meta-Llama-3-8B-Instruct --enable-offline-docs
Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure.
Key capabilities:
The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: examples/ray_serving/ray_serve_deepseek.py.
Learn more about Ray Serve LLM with the official Ray Serve LLM documentation.