Custom models

The built-in guards cover checks that most applications share — personal data, topics, prompt injection. When you need something specific to your product, fine-tune your own model: give Opik a labeled dataset and a description of what to catch, and it trains a language model into a check that answers one yes-or-no question about a piece of text, such as "does this reply read as off-brand?" or "is this abusive?".

Opik runs the whole loop for you. There is no training infrastructure to set up and nothing to deploy: you supply the examples, Opik fine-tunes the model on its GPUs, reports how accurate it turned out, and serves it inline from that point on. You can do it from the UI or from the SDK.

Once created, a model behaves like any other guard: select it in the custom classifier guard of a policy, or reference it by name from code.

When to fine-tune

A fine-tuned model is not the only way to cover a check the built-in guards miss, and it is not always the right one:

• An LLM judge needs no model of your own — you write the rule in plain language and a model from your workspace's providers judges it. Fastest to get running, most expensive per call.
• A manual guardrail suits a one-off heuristic or a business rule that is genuinely just code, like a list of competitor names.
• A fine-tuned model is what those two graduate into. Once a check has to run on every request and be right, training a small model on your labels is both cheaper per call than prompting a general model and more accurate than a heuristic.

Fine-tuned or prompt-only

Opik can also stand a model up from a description alone, with no training at all. The two options differ in what you prepare and in what you get back.

Reach for fine-tuned whenever the check matters: a trained model is faster and cheaper to run at high volume than prompting a general model on every request, it learns the edge cases your labels encode, and it comes with a number you can judge before trusting it.

Prompt only is the cheap first draft. Creating one costs nothing, so it is a quick way to confirm your description says what you meant — and to collect the disagreements that become your training labels — before booking GPU minutes on it.

Create a model in the UI

Open Guardrails in the sidebar, switch to the Models tab, and choose Create model. The tab lists every model available to the workspace, including Opik's own built-in detectors, so you can tell at a glance which are prompt-only, which are fine-tuned, and which came with the platform.

<Warning>
  A model cannot be renamed. Policies reference it by this name, so choose one you can live with.
</Warning>

This text is the model's prompt: it is what the model judges with, both while training and at inference time. Be specific about the edge you care about, since that is the part a short description usually leaves ambiguous.

If you pick **Prompt only** instead, there is no dataset. You can add examples it should flag and examples it should let through — one per line, both optional — which are folded into the prompt and are usually the quickest way to fix a model that is judging the wrong edge.

The training dataset

Fine-tuning takes a .jsonl file — one JSON object per line — of up to 50 MB:

{"text": "Get lost, nobody wants you around here.", "label": true}
{"text": "Thanks so much for your help today!", "label": false}

Label a row true when the text violates the guardrail and false when it does not.

While it trains, and if it fails

Training takes GPU minutes and cannot be stopped once it starts. The Models tab shows the model's status as it goes, and a model cannot be deleted until the job has finished or failed.

If training fails, it cannot be retried — check that the dataset has enough rows in both classes and no conflicting labels, then create a new model from the corrected file. The row menu's Reuse settings action reopens the form with the name and description prefilled, so you only need to attach the file again.

Inspect a model

Opening a model shows its status and type, the description you gave it, and — for a fine-tuned model — what the training job produced: how many rows it learned from, how many epochs it ran, the final training loss, and its accuracy on data held back from training.

Held-out accuracy is the number to look at. It is the only one measured on text the model has not seen, which makes it the only one that says anything about production. Opik does not block a weak model from being used, so this judgement is yours to make: if accuracy is low, the usual causes are too few examples, an unbalanced set, or a description that does not match how the rows were actually labeled.

The System prompt section shows the exact text the model judges with, built from your description. It is worth expanding — a model that flags the wrong thing is usually a prompt that says something slightly different from what you meant, and this is where you can see it.

The panel also lists which policies currently use the model, so you can see what you would affect before deleting it.

Use it in a guardrail

Add the model to a policy: switch on the custom classifier guard, select your model, and set a threshold. Only models that are ready can be selected.

The guard fails when the model's score is at or above the threshold. Lower it to be stricter, raise it to be more permissive.

From then on, any application that references the policy runs your model along with the policy's other checks — no application change needed to roll it out:

from opik.guardrails import Guardrail

guardrail = Guardrail.from_stored_policies(names=["support-tone"])

Fine-tune from the SDK

Fine-tuning is also a single SDK call, which is what you want when the dataset lives in version control or the training runs as part of a pipeline. Pass your labeled examples to create_custom_guardrail and Opik trains the model and serves it under the name you chose.

Each example is text plus a binary label, where 1 means the check holds and the guardrail should fail. The description completes the sentence "Determine whether it ...".

from opik.guardrails import create_custom_guardrail

result = create_custom_guardrail(
    name="toxicity-v1",
    description="contains toxic or abusive language",
    examples=[
        {"text": "Get lost, nobody wants you around here.", "label": 1},
        {"text": "Thanks so much for your help today!", "label": 0},
        # ... more labeled examples
    ],
)

print(result["eval_metrics"])

By default the call blocks until training finishes and returns evaluation metrics on held-out validation and test splits, so you can judge quality before using the model. Pass wait=False to return immediately instead, and overwrite=True to retrain under a name that already exists.

```python
def on_progress(status):
    p = status.get("progress", {})
    print(
        f"{status['status']} {p.get('percent', 0)}% "
        f"epoch={p.get('epoch')} "
        f"val_f1={p.get('latest_eval', {}).get('eval_validation_f1')}"
    )

create_custom_guardrail(
    name="toxicity-v1",
    description="contains toxic or abusive language",
    examples=[...],
    callback=on_progress,
)
```

Next steps