provider-mlflow-gateway (MLflow AI Gateway)

This example demonstrates how to use MLflow AI Gateway as an LLM provider in promptfoo.

To get started:

bash

npx promptfoo@latest init --example provider-mlflow-gateway

Setup

Install and start MLflow:

bash

pip install mlflow[genai]
mlflow server --host 127.0.0.1 --port 5000

Create a gateway endpoint in the MLflow UI at http://localhost:5000 (AI Gateway → Create Endpoint).
Set environment variables:

bash

export MLFLOW_GATEWAY_URL=http://localhost:5000

Run the evaluation:

bash

promptfoo eval -c promptfooconfig.yaml

Configuration

Update my-chat-endpoint in promptfooconfig.yaml with the name of the gateway endpoint you created. The example also uses that endpoint as the llm-rubric grader, so it runs without a separate OpenAI API key.

See the MLflow Gateway provider docs for all configuration options.