README

PhotoPrism — Vision Package

Last Updated: March 3, 2026

Overview

internal/ai/vision provides the shared model registry, request builders, and parsers that power PhotoPrism’s caption, label, face, NSFW, and future generate workflows. It reads vision.yml, normalizes models, and dispatches calls to one of three engines:

• TensorFlow (built‑in) — default Nasnet / NSFW / Facenet models, no remote service required. Long-running TensorFlow inference can accumulate C-allocated tensor memory until GC finalizers run, so PhotoPrism periodically triggers garbage collection to return that memory to the OS; tune with PHOTOPRISM_TF_GC_EVERY (default 200, 0 disables). Lower values reduce peak RSS but increase GC overhead and can slow indexing, so keep the default unless memory pressure is severe.
• Ollama — local or proxied multimodal LLMs. See ollama/README.md for tuning and schema details. The engine defaults to ${OLLAMA_BASE_URL:-http://ollama:11434}/api/generate, trimming any trailing slash on the base URL; set OLLAMA_BASE_URL=https://ollama.com to opt into cloud defaults.
• OpenAI — cloud Responses API. See openai/README.md for prompts, schema variants, and header requirements.

Configuration

Models

The vision.yml file is usually kept in the storage/config directory (override with PHOTOPRISM_VISION_YAML). It defines a list of models under Models:. Key fields are captured below. If a type is omitted entirely, PhotoPrism will auto-append the built-in defaults (labels, nsfw, face, caption) so you no longer need placeholder stanzas. The Thresholds block is optional; missing or out-of-range values fall back to defaults.

Run Modes

Note: For performance reasons, on-index is only supported for the built-in TensorFlow models.

Model Options

The model Options adjust model parameters such as temperature, top-p, and schema constraints when using Ollama or OpenAI. Rows are ordered exactly as defined in vision/model_options.go.

Model Service

Configures the endpoint URL, method, format, and authentication for Ollama, OpenAI, and other engines that perform remote HTTP requests:

Authentication: All credentials and identifiers support ${ENV_VAR} expansion. Service.Key sets Authorization: Bearer <token>; Username/Password injects HTTP basic authentication into the service URI when it is not already present. When Service.Key is empty, PhotoPrism defaults to OPENAI_API_KEY (OpenAI engine) or OLLAMA_API_KEY (Ollama engine), also honoring their _FILE counterparts. Key and schema file paths must reference readable regular files (directories are ignored/rejected).

Field Behavior & Precedence

• Model identifier resolution order: Service.Model → Model → Name. Model.GetModel() returns (id, name, version) where Ollama receives name:version and other engines receive name plus a separate Version.
• Env expansion runs for all Service credentials and Model overrides; empty or disabled models return empty identifiers.
• Options merging: engine defaults fill missing fields; explicit values always win. Temperature is capped at MaxTemperature.
• Authentication: Service.Key sets Authorization: Bearer <token>; Username/Password inject HTTP basic auth into the service URI when not already present.
• Reasoning control: Service.Think maps to ApiRequest.Think and is serialized only when non-empty (omitempty). During JSON encoding, "true" / "false" are converted to boolean true / false; other non-empty values are sent as strings.

Minimal Examples

TensorFlow (built‑in defaults)

Models:
  - Type: labels
    Default: true
    Run: auto

  - Type: nsfw
    Default: true
    Run: auto

  - Type: face
    Default: true
    Run: auto

Ollama Labels

Models:
  - Type: labels
    Model: gemma3:latest
    Engine: ollama
    Run: newly-indexed
    Service:
      Uri: ${OLLAMA_BASE_URL}/api/generate

More Ollama guidance: internal/ai/vision/ollama/README.md.

OpenAI Captions

Models:
  - Type: caption
    Model: gpt-5-mini
    Engine: openai
    Run: newly-indexed
    Service:
      Uri: https://api.openai.com/v1/responses
      Org: ${OPENAI_ORG}
      Project: ${OPENAI_PROJECT}
      Key: ${OPENAI_API_KEY}

More OpenAI guidance: internal/ai/vision/openai/README.md.

Custom TensorFlow Labels (SavedModel)

Models:
  - Type: labels
    Name: transformer
    Engine: tensorflow
    Path: transformer   # resolved under assets/models
    Resolution: 224     # keep standard TF input size unless your model differs
    TensorFlow:
      Output:
        Logits: true    # set true for most TF2 SavedModel classifiers

Custom TensorFlow Models — What’s Supported

• Scope: Classification tasks only (labels). TensorFlow models cannot generate captions today; use Ollama or OpenAI for captions.
• Location & paths: If Path is empty, the model is loaded from assets/models/<name> (lowercased, underscores). If Path is set, it is still searched under assets/models; absolute paths are not supported.
• Expected files: saved_model.pb, a variables/ directory, and a labels.txt alongside the model; use TF2 SavedModel classifiers.
• Resolution: Stays at 224px unless your model requires a different input size; adjust Resolution and the TensorFlow.Input block if needed.
• Sources: Labels produced by TensorFlow models are recorded with source image; overriding the source isn’t supported yet.
• Config file: vision.yml is the conventional name; in the latest version, .yaml is also supported by the loader.

CLI Quick Reference

• List models: photoprism vision ls (shows resolved IDs, engines, options, run mode, disabled flag).
• Run a model: photoprism vision run -m labels --count 5 (use --force to bypass Run rules).
• Validate config: photoprism vision ls --json to confirm env-expanded values without triggering calls.

When to Choose Each Engine

• TensorFlow: fast, offline defaults for core features (labels, faces, NSFW). Zero external deps.
• Ollama: private, GPU/CPU-hosted multimodal LLMs; best for richer captions/labels without cloud traffic.
• OpenAI: highest quality reasoning and multimodal support; requires API key and network access.

Model Unload on Idle

PhotoPrism currently keeps TensorFlow models resident for the lifetime of the process to avoid repeated load costs. A future “model unload on idle” mode would track last-use timestamps and close the TensorFlow session/graph after a configurable idle period, releasing the model’s memory footprint back to the OS. The trade-off is higher latency and CPU overhead when a model is used again, plus extra I/O to reload weights. This may be attractive for low-frequency or memory-constrained deployments but would slow continuous indexing jobs, so it is not enabled today.

Troubleshooting

• If face model initialization fails with Read less bytes than requested (often followed by invalid face model configuration in GenerateFaceEmbeddings tests), reinstall the local FaceNet assets:
  • rm -f /tmp/photoprism/facenet.zip
  • rm -rf assets/models/facenet
  • make dep-tensorflow (or scripts/download-facenet.sh)
  • Re-run: go test ./internal/ai/face -run TestNet -count=1 and go test ./internal/ai/vision -run TestGenerateFaceEmbeddings -count=1

Related Docs

• Ollama specifics: internal/ai/vision/ollama/README.md
• OpenAI specifics: internal/ai/vision/openai/README.md
• REST API reference: https://docs.photoprism.dev/
• Developer guide (Vision): https://docs.photoprism.app/developer-guide/api/