Annotate Traces - Opik

Building Feedback Loops with Trace Annotations

This video demonstrates how to add human feedback and annotations to your LLM traces, creating a powerful feedback loop for continuous improvement. You'll learn to mark traces as high or low quality, add specific feedback scores, and build datasets of annotated examples for future evaluation and fine-tuning. Annotations bridge the gap between raw trace data and actionable insights for model improvement.

Key Highlights

Easy Feedback Addition: Use the feedback scores section and pen icon to quickly add human annotations directly to traces in the Opik UI
Custom Feedback Definitions: Create categorical or numerical feedback metrics in Configuration → Feedback Definitions to match your specific evaluation needs
Flexible Scoring Options: Define custom categories (pass/fail, quality ratings) or sliding scales (0-1) based on metrics like accuracy, relevance, or tone
Smart Filtering: Filter traces by feedback scores to quickly identify high-performing or problematic outputs for targeted analysis
Dataset Creation: Convert annotated traces directly into datasets for training data collection and model improvement workflows
Built-in Metrics Library: Access pre-built metrics like answer relevance, Levenshtein distance, and hallucination detection out of the box
LLM as a Judge: Use automated LLM-based evaluation where a second LLM evaluates responses from your primary model
Pattern Identification: Annotations help identify performance patterns, create training data, and establish feedback loops between users and developers