docs/en/run/cli/local-server.mdx
import Feedback from "/snippets/page-feedback.mdx";
GenieX includes a built-in inference server that exposes an OpenAI-compatible API. Run models on-device and connect them to any application or framework that speaks the OpenAI protocol — agentic frameworks like LangChain, AI-native apps like OpenClaw, or your own code. No cloud dependency.
geniex serve does not auto-download models.Pull a model:
geniex pull ai-hub-models/Qwen3-4B-Instruct-2507
Start the server:
geniex serve
The server runs on http://127.0.0.1:18181 by default. Keep this terminal open and make requests from another one. Run geniex serve -h for all configurable options.
Creates a model response for a conversation. Supports LLM (text-only) and VLM (image + text).
{
"model": "ai-hub-models/Qwen3-4B-Instruct-2507",
"messages": [
{"role": "user", "content": "Hello! Briefly introduce yourself."}
],
"max_tokens": 256,
"temperature": 0.7,
"stream": false
}
Open http://127.0.0.1:18181 in your browser to access the built-in Swagger UI.
Step 1. Expand the POST /v1/chat/completions endpoint to view the example request body and schema.
Step 2. Click Try it out, edit the request body as needed, then click Execute.
Step 3. View the response — a 200 status with the model's generated reply.
image_url.url accepts three formats:
| Format | Example |
|---|---|
Local file path (the file:// prefix is optional) | C:/Users/Username/Pictures/photo.jpg, file:///tmp/photo.jpg |
| HTTP / HTTPS URL — fetched by the server | https://example.com/image.jpg |
| Base64 data URL — inline image bytes | data:image/png;base64,iVBORw0KGgo... |
{
"model": "ai-hub-models/Qwen2.5-VL-7B-Instruct",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image succinctly."},
{"type": "image_url", "image_url": {"url": "</path/to/image>"}}
]
}
]
}
In Swagger UI, replace the request body with this VLM payload, point image_url.url to a local image, then click Execute.
Because the server speaks the OpenAI protocol, you can point the official openai Python client at the local endpoint and reuse any existing OpenAI code. Install with pip install openai, then create a client:
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:18181/v1",
api_key="geniex", # any non-empty string; the server does not check it
)
The examples below reuse this client. Replace the model value with a model you have already pulled. The optional :<precision> suffix (e.g. Q4_0, Q4_K_M, Q8_0) selects a quantization variant — Q4_0 is recommended for llama.cpp on Hexagon NPU. See Precisions (Quantizations) Supported.
Print each delta as it arrives:
stream = client.chat.completions.create(
model="unsloth/Qwen3-4B-GGUF:Q4_0",
messages=[
{"role": "user", "content": "Hello! Briefly introduce yourself."},
],
max_tokens=256,
temperature=0.7,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print()
Single request, single response, no streaming — the standard OpenAI chat.completions.create shape. The enable_think=False extra parameter turns off Qwen3's default <think>…</think> reasoning prefix so the reply content stays clean.
resp = client.chat.completions.create(
model="unsloth/Qwen3-4B-GGUF:Q4_0",
messages=[
{"role": "user", "content": "Hello! Briefly introduce yourself."},
],
max_tokens=128,
temperature=0.7,
extra_body={"enable_think": False},
)
print(resp.choices[0].message.content)
print("finish_reason:", resp.choices[0].finish_reason)
print("usage:", resp.usage)
Output:
Hello! I'm Qwen, a large language model developed by Alibaba Cloud. I can help with a wide range of tasks, including answering questions, writing articles, creating stories, and more. I'm here to assist you in any way I can! How can I help you today?
finish_reason: stop
usage: CompletionUsage(completion_tokens=58, prompt_tokens=19, total_tokens=77, ...)
Function/tool calling uses the standard OpenAI tools schema. The server extracts the tool call from the model's generated text (<tool_call>…</tool_call> tags or a fenced ```json block) and re-emits it as OpenAI tool_calls. The flow works with VLMs too — the model can look at an image, decide what to search for, and call a tool.
The example below walks through a two-step agentic loop with qualcomm/Qwen3-VL-4B-Instruct: (1) the VLM identifies a landmark from a photo and calls web_search, (2) you execute the search locally and feed the results back so the VLM writes a grounded reply.
Install the search library used by the tool (pip install ddgs — DuckDuckGo, no API key required), then:
import json
from ddgs import DDGS
tools = [
{
"type": "function",
"function": {
"name": "web_search",
"description": "Search the web for travel information about a location.",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query, e.g. 'things to do in Kyoto'"},
},
"required": ["query"],
},
},
}
]
def web_search(query: str) -> list[dict]:
return [
{"title": r["title"], "snippet": r["body"], "url": r["href"]}
for r in DDGS().text(query, max_results=3)
]
IMAGE_URL = "https://images.pexels.com/photos/402028/pexels-photo-402028.jpeg?w=1024"
system_prompt = (
"You are a travel assistant. Call the web_search tool to look up any location "
"the user asks about before answering. Once you receive the tool results, do not "
"call the tool again - use them to write a short, friendly reply for the user. "
"Emit tool calls in the exact format: "
'<tool_call>{"name": "web_search", "arguments": {"query": "<your query>"}}</tool_call>'
)
messages = [
{"role": "system", "content": system_prompt},
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": IMAGE_URL}},
{"type": "text", "text": "Identify the landmark, then call web_search for travel tips about it."},
],
},
]
# Step 1 - VLM identifies the landmark and requests a web_search call.
first = client.chat.completions.create(
model="qualcomm/Qwen3-VL-4B-Instruct",
messages=messages,
tools=tools,
tool_choice="auto",
max_tokens=512,
extra_body={"enable_think": False},
)
call = first.choices[0].message.tool_calls[0]
print("finish_reason:", first.choices[0].finish_reason) # -> "tool_calls"
print("call:", call.function.name, call.function.arguments)
# Step 2 - run the tool, feed the result back as a fresh text-only conversation.
result = web_search(**json.loads(call.function.arguments))
followup = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": "Summarize travel tips using the search results below."},
first.choices[0].message,
{"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)},
]
final = client.chat.completions.create(
model="qualcomm/Qwen3-VL-4B-Instruct",
messages=followup,
max_tokens=256,
extra_body={"enable_think": False},
)
print(final.choices[0].message.content)
Output (grounded on the live DuckDuckGo results — the exact prose varies with what the web returns that day):
finish_reason: tool_calls
call: web_search {"query": "Kiyomizu-dera travel tips"}
Here's a quick summary of travel tips for Kiyomizu-dera in Kyoto:
**1. Best Times to Visit:**
Early morning (around 7-8 AM) or late afternoon (4-6 PM) are ideal to avoid the crowds.
Peak hours (11 AM–2 PM) can be very busy, so plan accordingly.
**2. How to Get There:**
Take the Kiyomizu-dera train or bus from Kyoto Station to the temple.
The temple is located on a hill with a wooden stage built without nails, and the walkways are accessible, though narrow.
**3. What to See:**
- The great wooden stage (without nails)
- Otawa waterfall and its three streams
- Jishu love shrine
- The Sannenzaka and Ninenzaka approach streets
- Night illuminations (best experienced at night)
...
GET /v1/models — list available models.GET /v1/models/{model} — get info about a specific model.