docs/docs/genai/concepts/feedback.mdx
import { APILink } from "@site/src/components/APILink"; import ImageBox from "@site/src/components/ImageBox"; import FeatureHighlights from "@site/src/components/FeatureHighlights"; import TilesGrid from "@site/src/components/TilesGrid"; import TileCard from "@site/src/components/TileCard"; import { ClipboardCheck, UserCheck, Bot, UsersRound, MessageSquare, FileText, Target } from "lucide-react"; import FeedbackArchitectureImageUrl from '@site/static/images/llms/tracing/schema/feedback_architecture.png';
Feedback in MLflow represents the result of any quality assessment performed on your LLM application or AI agent outputs. It provides a standardized way to capture evaluations, whether they come from automated systems, LLM judges, or human reviewers.
Feedback serves as the bridge between running your application and understanding its quality, enabling you to systematically track performance across different dimensions like correctness, relevance, safety, and adherence to guidelines.
<ImageBox src="/images/llms/tracing/genai-human-feedback.png" alt="Feedbacks attached to traces" width="70%" caption="Feedbacks attached to traces" />
<FeatureHighlights features={[ { icon: ClipboardCheck, title: "Manual Quality Checks", description: "Manual quality checks are important for ensuring the quality of your LLM application or AI agent. For example, you can attach a feedback to indicate the hallucination in the response and compare quality between different models." }, { icon: UserCheck, title: "End-User Feedbacks", description: "Feedbacks from end-users are precious for improving the quality of your LLM application or AI agent. By storing feedbacks on your traces, you can easily monitor the user satisfaction of your application over time." }, { icon: Bot, title: "LLM Judge Evaluation", description: "LLM judges are powerful tools for systematically running quality checks at scale. When using MLflow's evaluation framework, feedbacks from LLM judges are attached to the traces, enabling you to track evaluation results in the unified way as manual quality checks." }, { icon: UsersRound, title: "Collaborative Annotation", description: "Quality checks are often performed by multiple annotators to ensure the robustness of the output. MLflow tracks metadata and revision history of the feedbacks and supports aggregation of feedbacks from multiple annotators." } ]} />
Feedback is often created by different sources, such as human annotators, LLM judges, or real user's feedback in the application. The <APILink fn="mlflow.entities.Feedback">Feedback</APILink> object in MLflow is a standard container for storing these signals along with metadata to track
how they are created. Feedbacks are associated with a Trace, or a particular Span in the Trace.
| Field | Type | Description |
|---|---|---|
name | str | A string identifying the specific quality aspect being assessed |
value | Any | The actual feedback value, which can be |
Human Feedback for Hallucination in the Response
{
"name": "hallucination",
"value": false,
"rationale": "The response is factual and does not contain any hallucinations.",
"source": {
"source_type": "HUMAN",
"source_id": "[email protected]"
}
}
LLM Judge Feedback for Factual Accuracy
{
"name": "factual_accuracy",
"value": 0.85,
"rationale": "The response correctly identifies 3 out of 4 key facts about MLflow, but incorrectly states the founding year.",
"source": {
"source_type": "LLM_JUDGE",
"source_id": "openai:/4o-mini"
},
"metadata": {
# Store link to the prompt used for the judge, registered in MLflow Prompt Registry
"judge_prompt": "prompts:factual_accuracy_judge/1"
}
}
Error Feedback from LLM Judge (Rate Limit Exceeded)
{
"name": "safety",
"error": {
"error_code": "RATE_LIMIT_EXCEEDED",
"error_message": "Rate limit for the judge exceeded.",
"stack_trace": "..."
},
"source": {
"source_type": "LLM_JUDGE",
"source_id": "openai:/4o-mini"
}
}