docs/providers/inferrs.md
inferrs serves local models behind an OpenAI-compatible /v1 API. OpenClaw talks to it through the generic openai-completions adapter.
| Property | Value |
|---|---|
| Provider id | inferrs (custom; configure under models.providers.inferrs) |
| Plugin | none — not a bundled OpenClaw provider plugin |
| Auth env var | none required; any value works if your inferrs server has no auth |
| API | OpenAI-compatible (openai-completions) |
| Suggested base URL | http://127.0.0.1:8080/v1 (or wherever your inferrs server listens) |
Gemma 4 on a local inferrs server:
{
agents: {
defaults: {
model: { primary: "inferrs/google/gemma-4-E2B-it" },
models: {
"inferrs/google/gemma-4-E2B-it": {
alias: "Gemma 4 (inferrs)",
},
},
},
},
models: {
mode: "merge",
providers: {
inferrs: {
baseUrl: "http://127.0.0.1:8080/v1",
apiKey: "inferrs-local",
api: "openai-completions",
models: [
{
id: "google/gemma-4-E2B-it",
name: "Gemma 4 E2B (inferrs)",
reasoning: false,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 131072,
maxTokens: 4096,
compat: {
requiresStringContent: true,
},
},
],
},
},
},
}
OpenClaw can start inferrs itself only when an inferrs/... model is selected. Add localService to the same provider entry:
{
models: {
providers: {
inferrs: {
baseUrl: "http://127.0.0.1:8080/v1",
apiKey: "inferrs-local",
api: "openai-completions",
timeoutSeconds: 300,
localService: {
command: "/opt/homebrew/bin/inferrs",
args: [
"serve",
"google/gemma-4-E2B-it",
"--host",
"127.0.0.1",
"--port",
"8080",
"--device",
"metal",
],
healthUrl: "http://127.0.0.1:8080/v1/models",
readyTimeoutMs: 180000,
idleStopMs: 0,
},
models: [
{
id: "google/gemma-4-E2B-it",
name: "Gemma 4 E2B (inferrs)",
reasoning: false,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 131072,
maxTokens: 4096,
compat: {
requiresStringContent: true,
},
},
],
},
},
},
}
command must be an absolute path. Run which inferrs on the Gateway host and use that path. Full field reference: Local model services.
<Warning>
If OpenClaw runs fail with:
```text
messages[1].content: invalid type: sequence, expected a string
```
set `compat.requiresStringContent: true` in the model entry. OpenClaw then flattens pure text content parts into plain strings before sending the request.
</Warning>
```json5
compat: {
requiresStringContent: true,
supportsTools: false
}
```
That reduces prompt pressure on stricter local backends. If tiny direct requests still work but normal OpenClaw agent turns keep crashing inside `inferrs`, treat it as an upstream model/server limitation rather than an OpenClaw transport issue.
```bash
curl http://127.0.0.1:8080/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model":"google/gemma-4-E2B-it","messages":[{"role":"user","content":"What is 2 + 2?"}],"stream":false}'
```
```bash
openclaw infer model run \
--model inferrs/google/gemma-4-E2B-it \
--prompt "What is 2 + 2? Reply with one short sentence." \
--json
```
If the first command works but the second fails, see Troubleshooting below.