docs/docs/genai/eval-monitor/legacy-llm-evaluation.mdx
import { APILink } from "@site/src/components/APILink"; import useBaseUrl from '@docusaurus/useBaseUrl'; import ImageBox from "@site/src/components/ImageBox"; import WorkflowSteps from '@site/src/components/WorkflowSteps'; import { Bot, Database, Target, Play } from 'lucide-react';
:::info
This is a migration guide for users who are using the legacy LLM evaluation capability through mlflow.evaluate API and see the following warning while migrating to MLflow 3.
The mlflow.evaluate API has been deprecated as of MLflow 3.0.0.
If you are new to MLflow or its evaluation capabilities, start from the <ins>MLflow 3 Evaluation</ins> guide instead.
:::
MLflow 3 introduces a new evaluation suite that is optimized for evaluating LLM applications and AI agents. Compared to the legacy evaluation through the mlflow.evaluate API, the new suite offers the following benefits:
MLflow 3 displays the evaluation results with intuitive visualizations. Each prediction is recorded with a trace, which allows you to further investigate the result in details and identify the root cause of low quality predictions.
<table> <tr> <th>Old Results</th> <td><ImageBox src="/images/mlflow-3/eval-monitor/legacy-eval-result.png" alt="Legacy Evaluation" width="80%"/></td> </tr> <tr> <th>New Results</th> <td><ImageBox src="/images/mlflow-3/eval-monitor/prompt-evaluation-compare.png" alt="New Evaluation" width="80%"/></td> </tr> </table>A rich set of built-in LLM Judges and a flexible toolset to build your own LLM-as-a-Judge supports you to evaluate various aspects of your LLM applications. Furthermore, the new Agents-as-a-Judge capability evaluates complex trace with minimum context window consumption and boilerplate code.
The new evaluation suite is tightly integrated with other MLflow capabilities, such as tracing, prompt management, prompt optimization, making it an end-to-end solution for building high-quality LLM applications.
MLflow is rapidly evolving (changelog) and will continue strengthening its evaluation capabilities with the north star of Deliver production-ready AI. Migrating your workload to the new evaluation suite will ensure you have instant access to the latest and greatest features.
<WorkflowSteps width="wide" steps={[ { icon: Bot, title: "Wrap your model in a function", description: "If you are evaluating an MLflow Model, wrap the model in a function and pass it to the new evaluation API." }, { icon: Database, title: "Update dataset format", description: "Update the inputs and ground truth format to match the new evaluation dataset format." }, { icon: Target, title: "Migrate metrics", description: "Update the metrics to use the new built-in or custom scorers offered by MLflow 3." }, { icon: Play, title: "Run evaluation", description: "Execute the evaluation and make sure the results are as expected." } ]} />
:::tip[Before you start the migration]
Before starting the migration, we highly recommend you to visit the <ins>Evaluation Guide</ins> and go through the <ins>Quickstart</ins> to get a sense of the new evaluation suite. Basic understanding of the concepts will help you to migrate your existing workload smoothly.
:::
The old evaluation API accepts MLflow model URI as an evaluation target. The new evaluation API accepts a callable function as predict_fn argument instead, to provide more flexibility and control. This also eliminates the need of logging the model in MLflow before evaluation.
mlflow.evaluate(model=logged_model_info.model_uri, ...)
}</code> </pre> </td> <td style={{ maxWidth: '50%', verticalAlign: 'top' }}> <pre style={{ whiteSpace: 'pre', overflowX: 'auto', margin: 0 }}> <code className="language-python">{# Define a function that runs predictions.
def predict_fn(question: str) -> str:
response = openai.OpenAI().chat.completions.create(
model="gpt-5-mini",
messages=[
{"role": "system", "content": "Answer the following question in two sentences"},
{"role": "user", "content": question},
],
)
return response.choices[0].message.content
mlflow.genai.evaluate(predict_fn=predict_fn, ...) `}</code>
</pre> </td> </tr> </table>If you want to evaluate a pre-logged model with the new evaluation API, simply call the loaded model in the function.
# IMPORTANT: Load the model outside the predict_fn function. Otherwise the model will be loaded
# for each input in the dataset and significantly slow down the evaluation.
model = mlflow.pyfunc.load_model(model_uri)
def predict_fn(question: str) -> str:
return model.predict([question])[0]
The dataset format has been changed to be more flexible and consistent. The new format requirements are:
inputs: The input to the predict_fn function. The key(s) must match the parameter name of the predict_fn function.expectations: The expected output from the predict_fn function, namely, ground truth for the answer.outputs column or trace column to evaluate pre-generated outputs and traces.mlflow.evaluate(
data=eval_data, # Needed to specify the ground truth and prediction # columns name, otherwise MLflow does not recognize them.
targets="ground_truth",
predictions="predictions",
...
)
}</code> </pre> </td> <td style={{ maxWidth: '50%', verticalAlign: 'top' }}> <pre style={{ whiteSpace: 'pre', overflowX: 'auto', margin: 0 }}> <code className="language-python">{eval_data = [
{
"inputs": {"question": "What is MLflow?"},
"outputs": "MLflow is an open-source MLOps platform",
"expectations": {"answer": "MLflow is the largest open source AI engineering platform for agents and LLMs."},
},
{
"inputs": {"question": "What is Spark?"},
"outputs": "Apache Spark is an open-source distributed computing engine.",
"expectations": {"answer": "Apache Spark is an open-source, distributed computing system designed for big data processing and analytics."},
},
]
mlflow.genai.evaluate( data=eval_data, ... ) `}</code>
</pre> </td> </tr> </table>The new evaluation API supports a rich set of built-in and custom LLM-as-a-Judge metrics. The table below shows the mapping between the legacy metrics and the new metrics.
<table> <tr> <th>Metric</th> <th>Before</th> <th>After</th> </tr> <tr> <td>Latency</td> <td><APILink fn="mlflow.metrics.latency">`latency`</APILink></td> <td>Traces record latency and also span-level break down. You don't need to specify a metric to evaluate latency when running the new <APILink fn="mlflow.genai.evaluate" /> API.</td> </tr> <tr> <td>Token Count</td> <td><APILink fn="mlflow.metrics.token_count">`token_count`</APILink></td> <td>Traces record token count for LLM calls for most of popular LLM providers. For other cases, you can use a [custom scorer](/genai/eval-monitor/scorers/custom) to calculate the token count.</td> </tr> <tr> <td>Heuristic NLP metrics</td> <td><APILink fn="mlflow.metrics.toxicity">`toxicity`</APILink>, <APILink fn="mlflow.metrics.flesch_kincaid_grade_level">`flesch_kincaid_grade_level`</APILink>, <APILink fn="mlflow.metrics.ari_grade_level">`ari_grade_level`</APILink>, <APILink fn="mlflow.metrics.exact_match">`exact_match`</APILink>, <APILink fn="mlflow.metrics.rouge1">`rouge1`</APILink>, <APILink fn="mlflow.metrics.rouge2">`rouge2`</APILink>, <APILink fn="mlflow.metrics.rougeL">`rougeL`</APILink>, <APILink fn="mlflow.metrics.rougeLsum">`rougeLsum`</APILink></td> <td>Use a [Code-based Scorer](/genai/eval-monitor/scorers/custom) to implement the equivalent metrics. See the example below for reference.</td> </tr> <tr> <td>Retrieval metrics</td> <td><APILink fn="mlflow.metrics.precision_at_k">`precision_at_k`</APILink>, <APILink fn="mlflow.metrics.recall_at_k">`recall_at_k`</APILink>, <APILink fn="mlflow.metrics.ndcg_at_k">`ndcg_at_k`</APILink></td> <td>Use the new [built-in retrieval metrics](/genai/eval-monitor/scorers/llm-judge/predefined/#available-judges) or define a custom code-based scorer.</td> </tr> <tr> <td>Built-in LLM-as-a-Judge metrics</td> <td><APILink fn="mlflow.metrics.genai.answer_similarity">`answer_similarity`</APILink>, <APILink fn="mlflow.metrics.genai.answer_correctness">`answer_correctness`</APILink>, <APILink fn="mlflow.metrics.genai.answer_relevance">`answer_relevance`</APILink>, <APILink fn="mlflow.metrics.genai.relevance">`relevance`</APILink>, <APILink fn="mlflow.metrics.genai.faithfulness">`faithfulness`</APILink></td> <td>Use the new [built-in judges](/genai/eval-monitor/scorers/llm-judge/predefined/#available-judges). If the metric is not supported out of the box, define a custom LLM-as-a-Judge scorer using the <APILink fn="mlflow.genai.judges.make_judge">`make_judge`</APILink> API, following the example below.</td> </tr> <tr> <td>Custom LLM-as-a-Judge metrics</td> <td><APILink fn="mlflow.metrics.genai.make_genai_metric">`make_genai_metric`</APILink>, <APILink fn="mlflow.metrics.genai.make_genai_metric_from_prompt">`make_genai_metric_from_prompt`</APILink></td> <td>Use the <APILink fn="mlflow.genai.judges.make_judge">`make_judge`</APILink> API to define a custom LLM-as-a-Judge scorer, following the example below.</td> </tr> </table>The new evaluation API supports defining custom LLM-as-a-Judge metrics from a custom prompt template. This eliminates a lot of complexity and over-abstractions from the previous make_genai_metric and make_genai_metric_from_prompt APIs.
from mlflow.genai import make_judge
answer_similarity = make_judge(
name="answer_similarity",
instructions=(
"Evaluated on the degree of semantic similarity of the provided output to the expected answer.\n\n"
"Output: {{ outputs }}\n\n"
"Expected: {{ expectations }}"
),
feedback_value_type=int,
)
# Pass the scorer to the evaluation API.
mlflow.genai.evaluate(scorers=[answer_similarity, ...])
See the LLM Judges guide for more details.
Implementing a custom scorer for heuristic metrics is straightforward. You just need to define a function and decorate it with the <APILink fn="mlflow.genai.scorers.scorer">@scorer</APILink> decorator. The example below shows how to implement the exact_match metric.
@scorer
def exact_match(outputs: dict, expectations: dict) -> bool:
return outputs == expectations["expected_response"]
# Pass the scorer to the evaluation API.
mlflow.genai.evaluate(scorers=[exact_match, ...])
See the Code-based Scorers guide for more details.
Now you have migrated all components of the legacy evaluation API and are ready to run the evaluation!
mlflow.genai.evaluate(
data=eval_data,
predict_fn=predict_fn,
scorers=[answer_similarity, exact_match, ...],
)
To view the evaluation results, click the link in the console output, or navigate to the Evaluations tab in the MLflow UI.
<video src={useBaseUrl("/images/mlflow-3/eval-monitor/evaluation-result-video.mp4")} controls loop autoPlay muted aria-label="Prompt Evaluation" />
databricks:/<endpoint-name> as model provider, rather than endpoints:/<endpoint-name>Evaluations tab in the MLflow UI.model_type, targets, feature_names, env_manager, are removed in the new evaluation API.Please open an feature request in GitHub.
See MLflow 2 documentation for the legacy evaluation API.
It will likely be removed in MLflow 3.7.0 or a few releases after that.
No. The new evaluation suite is only for LLM applications and AI agents. If you are not building LLM applications or AI agents, you should use the <APILink fn="mlflow.models.evaluate" /> API, which offers perfect compatibility with mlflow.evaluate API but drops the LLM-specific features.