Back to Promptfoo

provider-mlflow-gateway (MLflow AI Gateway)

examples/provider-mlflow-gateway/README.md

0.121.131.0 KB
Original Source

provider-mlflow-gateway (MLflow AI Gateway)

This example demonstrates how to use MLflow AI Gateway as an LLM provider in promptfoo.

To get started:

bash
npx promptfoo@latest init --example provider-mlflow-gateway

Setup

  1. Install and start MLflow:
bash
pip install mlflow[genai]
mlflow server --host 127.0.0.1 --port 5000
  1. Create a gateway endpoint in the MLflow UI at http://localhost:5000 (AI Gateway → Create Endpoint).

  2. Set environment variables:

bash
export MLFLOW_GATEWAY_URL=http://localhost:5000
  1. Run the evaluation:
bash
promptfoo eval -c promptfooconfig.yaml

Configuration

Update my-chat-endpoint in promptfooconfig.yaml with the name of the gateway endpoint you created. The example also uses that endpoint as the llm-rubric grader, so it runs without a separate OpenAI API key.

See the MLflow Gateway provider docs for all configuration options.