Optimizer Frequently Asked Questions - Opik

Getting started

<AccordionGroup> <Accordion title="Why should I use the Opik Agent Optimizer?"> The Agent Optimizer provides a unified interface for optimizing **your existing prompts and agents** with cutting edge optimization algorithms. In addition to giving you access to cutting edge academic optimizers like GEPA, we also provide a set of algorithms that have been developed in-house based on production applications.

The optimizer will allow you to improve the performance of your agents without the need for
manual prompt engineering. You can also use the optimizer to reduce the size of the prompts in
your agents, reducing cost and latency while maintaining performance.

</Accordion> <Accordion title="How do I get started with the Optimizer ?"> To get started, you will need: 1. The prompt you want to optimize 2. A dataset of examples to optimizer on, you can start with as little as 10 3. A metric to evaluate the performance of the prompt

Once you have these, check out the Quickstart Guide to run your first optimization. </Accordion> <Accordion title="Can you help me optimize my prompt ?"> Yes, we would be more than happy to help you setup the Opik Optimizer for your use case ! You can join our Slack community and ask for help there. </Accordion> </AccordionGroup>

Optimization Algorithms

<AccordionGroup> <Accordion title="What are the supported optimization algorithms?"> Opik Agent Optimizer supports a wide range of optimization algorithms including: - [HRPO (Hierarchical Reflective Prompt Optimizer)](/development/optimization-runs/algorithms/hierarchical_adaptive_optimizer) - [GEPA Optimization](/development/optimization-runs/algorithms/gepa_optimizer) - [Evolutionary Optimization](/development/optimization-runs/algorithms/evolutionary_optimizer) - [Few-shot Bayesian Optimization](/development/optimization-runs/algorithms/fewshot_bayesian_optimizer) - [MetaPrompt Optimization](/development/optimization-runs/algorithms/metaprompt_optimizer)

If you would like us to add a new optimization algorithm, simply create an issue on our GitHub repository and we will be happy to add it ! </Accordion> <Accordion title="How do I choose the right optimizer for my task?"> Knowing which optimizer to use depends on your specific needs. As a rule of thumb, we recommend starting with HRPO (Hierarchical Reflective Prompt Optimizer) as this has been shown to be a strong baseline for most tasks.

You can also try to use:

GEPA: This is one of the top performing academic optimizers and is a good option if you have a complex task.
FewShotBayesianOptimizer: If you have a task that is quite repetitive in the formatting of the prompt and responses then this is a good option.

</Accordion> <Accordion title="How long should I run the optimization for?"> While some optimizations run for shorts period of time, it is common for optimizations to take a couple of hours to complete. As you are starting out, we recommend setting the `max_trials` parameter to a reasonable number and increasing / decreasing it as you go. </Accordion> <Accordion title="How many dataset samples do I need?"> The number of samples you need depends on your task. As a rule of thumb, we recommend starting with at least 10 samples. The more samples you have, the more accurate the optimization will be. </Accordion> <Accordion title="What models are supported for optimization?"> The Opik Agent Optimizer uses [LiteLLM](https://docs.litellm.ai/) to support 100+ LLM providers including: - **OpenAI** (GPT-4o, GPT-4o-mini, o1, o3-mini) - **Anthropic** (Claude 3.5 Sonnet, Claude 3 Opus) - **Google Gemini** (Gemini 2.0 Flash, Gemini 1.5 Pro) - **Azure OpenAI** - **Ollama** (local models like Llama 3, Mistral) - **OpenRouter** (access to multiple providers)

Models use the LiteLLM format: provider/model-name (e.g., gemini/gemini-2.0-flash).

<Tip> See the [Configure LLM Providers](/development/optimization-runs/optimization/configure_models) guide for detailed setup instructions and examples for each provider. </Tip> </Accordion> <Accordion title="The performance is not improving, what should I do?"> There are a few things you can try to improve the performance of your optimization: 1. Review your optimization metric, ideally it should provide the model with an insightful `reason` for the score it gives. By improving the quality of the metric, the optimizer will be able to make better optimizations. 2. Review your dataset, ideally it should be a diverse set of examples that cover the different scenarios you want to optimize for. 3. Use more powerful models for both the chatPrompt model and the optimizer, as models get more powerful, they will be able to generate better optimizations. </Accordion> </AccordionGroup>

Common Errors

For frequent setup and usage errors, head to Known Issues. We keep the error catalog there so fixes and version notes stay in one place.

Open challenges & advanced topics

<AccordionGroup> <Accordion title="Why do some optimized prompts look like gibberish?"> Research has shown "evil twin" prompts and unusual delimiters can perform well despite being hard to interpret. Optimizers explore the search space indiscriminately, so high-performing instructions aren't always human-readable. When interpretability matters, prefer algorithms like HRPO that include reasoning traces or enforce structure via custom metrics. </Accordion> <Accordion title="How expensive are optimizers in terms of API calls?"> Cost varies widely. Reflection-heavy optimizers (e.g., HRPO, GEPA) may call LLMs multiple times per trial, while MetaPrompt/Few-Shot Bayesian are lighter weight. Start with small `n_samples` and `max_trials`, monitor API usage, and review the [Benchmarks](/development/optimization-runs/algorithms/benchmarks) page for sample-efficiency notes. </Accordion> <Accordion title="Do optimized prompts generalize across tasks?"> Optimizers tune prompts for the dataset you provide. Prompts may overfit if the dataset lacks coverage. Use diverse datasets, consider chaining optimizers (e.g., Evolutionary → Few-Shot Bayesian) to encourage generalization, and re-evaluate on unseen samples before shipping. </Accordion> <Accordion title="How can I avoid overfitting during optimization?"> To prevent overfitting, use the `validation_dataset` parameter when calling `optimize_prompt()`. This separates your data into training (for generating prompt improvements) and validation (for evaluating candidate prompts). The optimizer will use the training dataset to understand failure modes and generate improvements, then rank candidates using the validation dataset. This mirrors standard ML practices and helps ensure your optimized prompt generalizes to unseen data.

```python
result = optimizer.optimize_prompt(
    prompt=my_prompt,
    dataset=training_dataset,  # Used for analysis and improvements
    validation_dataset=validation_dataset,  # Used for ranking candidates
    metric=my_metric,
    max_trials=5,
)
```

If you don't provide a validation dataset, the optimizer will use the same dataset for both training and validation, which may lead to overfitting on that specific dataset. For best results, split your data 70/30 or 80/20 between training and validation sets.

</Accordion> <Accordion title="Can I optimize for more than one objective (accuracy, cost, safety, etc.)?"> Yes—compose metrics using `MultiMetricObjective` or custom heuristics so optimizers weigh multiple goals. For complex trade-offs, capture human preferences in the metric reasons, or explore Pareto-aware optimizers like GEPA that surface trade-offs between accuracy and cost. </Accordion> <Accordion title="What about multimodal or agent workflows?"> Opik supports optimizing prompts that handle images/videos, as well as agents built with LangGraph, Google ADK, or MCP toolchains. Use the [Optimize agents](/development/optimization-runs/optimization/optimize_agents) and [Optimize multimodal](/development/optimization-runs/optimization/optimize_multimodal) guides for modality-specific advice. </Accordion> <Accordion title="How reliable are optimizer metrics?"> LLM-based metrics can be noisy. Always include deterministic checks when possible, and ensure your `ScoreResult.reason` is informative so reflective optimizers can identify true failure modes. See [Define metrics](/development/optimization-runs/optimization/define_metrics) and [Custom metrics](/development/optimization-runs/advanced/custom_metrics) for best practices. </Accordion> <Accordion title="Are there safety or ethical considerations?"> Yes. When optimizing prompts that handle sensitive content, bake alignment constraints into your dataset/metrics (e.g., moderation scores) and review outputs manually before deployment. Multi-objective setups help enforce safety alongside accuracy. </Accordion> </AccordionGroup>

Next Steps

Explore API Reference for detailed technical documentation.
Review the individual Optimizer pages under Optimization Algorithms.
Check out the Quickstart Guide.