How to Improve Ollama Tool-Calling Prompt Cache Reuse in nanobot

Some Ollama model templates move or remove tool definitions as a conversation switches between user, assistant, and tool messages. nanobot can send a correct append-only chat request while the model template still renders a different token prefix. On slower local hardware, re-evaluating that prefix can add tens of seconds to an otherwise simple tool-using turn.

This guide shows how to diagnose that specific pattern and create a derived llama3.1:8b tag with a prefix-stable tool template. It does not modify nanobot or overwrite the original Ollama model.

What you will build

• a repeatable two-turn cache check
• an optional derived llama3.1:8b-prefix-stable-v1 Ollama tag
• a nanobot model preset that uses the derived tag

When to use this

Use this guide when all of the following are true:

• direct Ollama responses are reasonably fast;
• nanobot becomes slow after the model calls a tool;
• Ollama logs show a long main prompt, a much shorter tool follow-up, and low initial cache reuse on the next main prompt;
• the model is llama3.1:8b with a template that renders concrete tools only for the final user message.

Do not apply this template to another model family without checking that model's tool-call format first.

Diagnose the rendered prompt

Stop any existing Ollama process, then start a single-slot debug server. A single slot makes the cache sequence easier to read.

macOS or Linux

OLLAMA_CONTEXT_LENGTH=16384 \
OLLAMA_NUM_PARALLEL=1 \
OLLAMA_DEBUG=1 \
ollama serve

Windows PowerShell

$env:OLLAMA_CONTEXT_LENGTH = "16384"
$env:OLLAMA_NUM_PARALLEL = "1"
$env:OLLAMA_DEBUG = "1"
ollama serve

In another terminal, use a fresh session and explicitly request a tool so both turns exercise the agent loop:

nanobot agent --session cli:ollama-cache-check \
  --message "Use the exec tool to calculate 2+2, then answer"
nanobot agent --session cli:ollama-cache-check \
  --message "Use the exec tool to calculate 4+7, then answer"

In the Ollama output, find each new prompt line and the first cached n_tokens line that follows it. Later increasing cached n_tokens lines are prompt-evaluation progress, not additional initial cache hits.

A cache-unfriendly tool template may produce a pattern like this:

turn 1 main:             2 / 8460 initially cached
turn 1 tool follow-up: 3713 / 3758 initially cached
turn 2 main:          3767 / 8519 initially cached

The cache is working, but the next main request can reuse only the shorter prompt. Hardware throughput determines how expensive the remaining evaluation is.

To inspect the API request bodies as well, add OLLAMA_DEBUG_LOG_REQUESTS=1 before starting Ollama. These logs can contain system prompts, workspace context, and user messages. Keep them local and disable request logging after diagnosis.

Why this happens with the stock template

The tested llama3.1:8b template conditionally expands the tool definitions inside a user message:

{{- if and $.Tools $last }}
  ... render tool definitions ...
{{- end }}

The first request ends with a user message, so the tools are rendered there. After nanobot appends an assistant tool call and its result, that user message is no longer last, so the same API request history renders without the concrete tool block. On the next user turn, the tools reappear at a new position.

This is a model-template behavior. At the API boundary, nanobot continues to append the assistant tool call and tool result and sends the same tool definitions.

Create a prefix-stable derived model

Create PrefixStable.Modelfile with the content below. The template keeps concrete tool definitions in the system block, where they remain in the same position across user and tool messages.

FROM llama3.1:8b

TEMPLATE """{{- if or .System .Tools }}<|start_header_id|>system<|end_header_id|>
{{- if .System }}

{{ .System }}
{{- end }}
{{- if .Tools }}

Cutting Knowledge Date: December 2023

When you receive a tool call response, use the output to format an answer to the original user question.

You are a helpful assistant with tool calling capabilities.

Given the following functions, respond with a JSON function call with the proper arguments when a tool is needed.

Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables.

{{ range .Tools }}
{{- . }}
{{ end }}
{{- end }}<|eot_id|>
{{- end }}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 }}
{{- if eq .Role "user" }}<|start_header_id|>user<|end_header_id|>

{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}
{{- else if eq .Role "assistant" }}<|start_header_id|>assistant<|end_header_id|>
{{- if .ToolCalls }}
{{ range .ToolCalls }}
{"name": "{{ .Function.Name }}", "parameters": {{ .Function.Arguments }}}{{ end }}
{{- else }}

{{ .Content }}
{{- end }}{{ if not $last }}<|eot_id|>{{ end }}
{{- else if eq .Role "tool" }}<|start_header_id|>ipython<|end_header_id|>

{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}
{{- end }}
{{- end }}"""

Create the new tag:

ollama create llama3.1:8b-prefix-stable-v1 -f PrefixStable.Modelfile
ollama list

Ollama reuses the existing model layers. The new tag adds a small template and manifest instead of copying the base weights.

Select the derived model in nanobot

Merge this preset into ~/.nanobot/config.json and select it:

{
  "providers": {
    "ollama": {
      "apiBase": "http://localhost:11434/v1"
    }
  },
  "modelPresets": {
    "ollamaPrefixStable": {
      "label": "Ollama Llama 3.1 prefix-stable",
      "provider": "ollama",
      "model": "llama3.1:8b-prefix-stable-v1",
      "maxTokens": 2048,
      "contextWindowTokens": 16384,
      "temperature": 0.1
    }
  },
  "agents": {
    "defaults": {
      "modelPreset": "ollamaPrefixStable"
    }
  }
}

Verify the selected model and repeat the two-turn check:

nanobot status
nanobot agent --session cli:ollama-stable-check \
  --message "Use the exec tool to calculate 2+2, then answer"
nanobot agent --session cli:ollama-stable-check \
  --message "Use the exec tool to calculate 4+7, then answer"

In one controlled test with Ollama 0.32.1, llama3.1:8b, and one slot, the second main request improved from 3767 / 8519 initially cached (44.22%) to 8505 / 8520 (99.82%). The number of re-evaluated tokens fell from 4752 to 15. Treat these numbers as a diagnostic example, not a performance guarantee.

Roll back

Switch agents.defaults.modelPreset back to the original preset. When no config uses the derived tag, remove it with:

ollama rm llama3.1:8b-prefix-stable-v1

Removing the derived tag does not remove llama3.1:8b.

Limitations

• The template above is specific to the tested llama3.1:8b tool-call format.
• Ollama or the model publisher may update the stock template in a later release.
• Validate multiple tool calls, tool errors, parallel calls, and long conversations before using a custom template for unattended workloads.
• A higher cache ratio reduces prompt evaluation, but model generation, tool execution, process startup, and storage can still dominate end-to-end latency.
• Multiple Ollama slots change cache scheduling and may produce different results.

Related nanobot docs

• Provider Cookbook: Ollama Local Model
• Providers and Models: Ollama
• Troubleshooting