docs/en/run/cli/local-server.mdx
import Feedback from "/snippets/page-feedback.mdx";
GenieX includes a built-in inference server that exposes an OpenAI-compatible API. Run models on-device and connect them to any application or framework that speaks the OpenAI protocol — agentic frameworks like LangChain, AI-native apps like OpenClaw, or your own code. No cloud dependency.
geniex serve does not auto-download models.Pull a model:
geniex pull ai-hub-models/Qwen3-4B-Instruct-2507
Start the server:
geniex serve
The server runs on http://127.0.0.1:18181 by default. Keep this terminal open and make requests from another one. Run geniex serve -h for all configurable options.
Creates a model response for a conversation. Supports LLM (text-only) and VLM (image + text).
{
"model": "ai-hub-models/Qwen3-4B-Instruct-2507",
"messages": [
{"role": "user", "content": "Hello! Briefly introduce yourself."}
],
"max_tokens": 256,
"temperature": 0.7,
"stream": false
}
Open http://127.0.0.1:18181 in your browser to access the built-in Swagger UI.
Step 1. Expand the POST /v1/chat/completions endpoint to view the example request body and schema.
Step 2. Click Try it out, edit the request body as needed, then click Execute.
Step 3. View the response — a 200 status with the model's generated reply.
image_url.url accepts three formats:
| Format | Example |
|---|---|
Local file path (the file:// prefix is optional) | C:/Users/Username/Pictures/photo.jpg, file:///tmp/photo.jpg |
| HTTP / HTTPS URL — fetched by the server | https://example.com/image.jpg |
| Base64 data URL — inline image bytes | data:image/png;base64,iVBORw0KGgo... |
{
"model": "ai-hub-models/Qwen2.5-VL-7B-Instruct",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image succinctly."},
{"type": "image_url", "image_url": {"url": "</path/to/image>"}}
]
}
]
}
In Swagger UI, replace the request body with this VLM payload, point image_url.url to a local image, then click Execute.
Because the server speaks the OpenAI protocol, you can point the official openai Python client at the local endpoint and reuse any existing OpenAI code. Install with pip install openai, then create a client:
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:18181/v1",
api_key="geniex", # any non-empty string; the server does not check it
)
The examples below reuse this client. Replace the model value with a model you have already pulled. The optional :<precision> suffix (e.g. Q4_0, Q4_K_M, Q8_0) selects a quantization variant — Q4_0 is recommended for llama.cpp on Hexagon NPU. See Precisions (Quantizations) Supported.
Print each delta as it arrives:
stream = client.chat.completions.create(
model="unsloth/Qwen3-4B-GGUF:Q4_0",
messages=[
{"role": "user", "content": "Hello! Briefly introduce yourself."},
],
max_tokens=256,
temperature=0.7,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print()
Single request, single response, no streaming — the standard OpenAI chat.completions.create shape. The enable_think=False extra parameter turns off Qwen3's default <think>…</think> reasoning prefix so the reply content stays clean.
resp = client.chat.completions.create(
model="unsloth/Qwen3-4B-GGUF:Q4_0",
messages=[
{"role": "user", "content": "Hello! Briefly introduce yourself."},
],
max_tokens=128,
temperature=0.7,
extra_body={"enable_think": False},
)
print(resp.choices[0].message.content)
print("finish_reason:", resp.choices[0].finish_reason)
print("usage:", resp.usage)
Output:
Hello! I'm Qwen, a large language model developed by Alibaba Cloud. I can help with a wide range of tasks, including answering questions, writing articles, creating stories, and more. I'm here to assist you in any way I can! How can I help you today?
finish_reason: stop
usage: CompletionUsage(completion_tokens=58, prompt_tokens=19, total_tokens=77, ...)
Function/tool calling uses the standard OpenAI tools schema. The server extracts the tool call from the model's generated text (<tool_call>…</tool_call> tags or a fenced ```json block, produced by chat templates such as Qwen3's) and re-emits it as OpenAI tool_calls.
Two-step round trip: (1) the model returns a tool_calls message, (2) you execute the tool locally and feed the result back as a role="tool" message so the model can produce the final answer.
import json
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather in a given city.",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "City name, e.g. Beijing"},
},
"required": ["city"],
},
},
}
]
def get_weather(city: str) -> dict:
# replace this with your real implementation
return {"city": city, "temperature_c": 24, "condition": "Sunny"}
messages = [
{"role": "user", "content": "What is the weather in Beijing? Use the tool."},
]
# Step 1 — model requests a tool call.
first = client.chat.completions.create(
model="unsloth/Qwen3-4B-GGUF:Q4_0",
messages=messages,
tools=tools,
tool_choice="auto",
max_tokens=256,
extra_body={"enable_think": False},
)
call = first.choices[0].message.tool_calls[0]
print("finish_reason:", first.choices[0].finish_reason) # -> "tool_calls"
print("call:", call.function.name, call.function.arguments)
# Step 2 — run the tool, feed the result back.
result = get_weather(**json.loads(call.function.arguments))
messages.append(first.choices[0].message)
messages.append(
{
"role": "tool",
"tool_call_id": call.id,
"content": json.dumps(result),
}
)
final = client.chat.completions.create(
model="unsloth/Qwen3-4B-GGUF:Q4_0",
messages=messages,
tools=tools,
max_tokens=128,
extra_body={"enable_think": False},
)
print(final.choices[0].message.content)
Output:
finish_reason: tool_calls
call: get_weather {"city": "Beijing"}
The weather in Beijing is 24°C and sunny.
GET /v1/models — list available models.GET /v1/models/{model} — get info about a specific model.