dspy.BetterTogether

BetterTogether is a meta-optimizer proposed in the paper Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together by Dilara Soylu, Christopher Potts, and Omar Khattab. It combines prompt optimization and weight optimization (fine-tuning) by applying them in a configurable sequence, allowing a student program to iteratively improve both its prompts and model parameters. The core insight is that prompt and weight optimization can complement each other: prompt optimization can potentially discover effective task decompositions and reasoning strategies, while weight optimization can specialize the model to execute these patterns more efficiently. Using these approaches together in sequences (e.g., prompt optimization then weight optimization) may allow each to build on the improvements made by the other.

::: dspy.BetterTogether handler: python options: members: - compile - get_params show_source: true show_root_heading: true heading_level: 2 docstring_style: google show_root_full_path: true show_object_full_path: false separate_signature: false inherited_members: true

How BetterTogether Works

BetterTogether executes optimizers in a configurable sequence, evaluating each intermediate result and returning the best performing program. Here's how it works:

1. Initialization with Custom Optimizers

When initialized, BetterTogether accepts any DSPy optimizers (Teleprompters) as keyword arguments. The keys become the optimizer names used in the strategy string:

optimizer = BetterTogether(
    metric=metric,
    p=GEPA(...),           # 'p' can be used in strategy
    w=BootstrapFinetune(...) # 'w' can be used in strategy
)

If no optimizers are provided, BetterTogether defaults to BootstrapFewShotWithRandomSearch (key: 'p') and BootstrapFinetune (key: 'w').

2. Strategy Execution

The strategy string defines the sequence of optimizers to apply. For example:

compiled = optimizer.compile(
    student,
    trainset=trainset,
    valset=valset,
    strategy="p -> w -> p"
)

This strategy "p -> w -> p" means:

1. Run prompt optimizer ('p')
2. Run weight optimizer ('w') on the result
3. Run prompt optimizer ('p') again on the result

At each step:

• The trainset is shuffled (if shuffle_trainset_between_steps=True)
• The optimizer is run on the current student program
• The result is evaluated on the validation set
• The candidate program and score are recorded

Since BetterTogether is a meta-optimizer, any sequence of optimizers can be combined. The optimizer names in the strategy string correspond to the keyword arguments from initialization. For example, you can sequence different prompt optimizers (note: this illustrates BetterTogether's flexibility, not necessarily a recommended configuration):

optimizer = BetterTogether(
    metric=metric,
    mipro=MIPROv2(metric=metric, auto="light"),
    gepa=GEPA(metric=metric, auto="light")
)

compiled = optimizer.compile(
    student,
    trainset=trainset,
    valset=valset,
    strategy="mipro -> gepa -> mipro"
)

3. Validation and Program Selection

BetterTogether can use a validation set in three ways:

• Explicit valset: If valset is provided, it's used for evaluation
• Auto-split: If valset_ratio > 0, a portion of trainset is held out for validation
• No validation: If both valset and valset_ratio are None/0, no validation occurs

After all optimization steps complete, the best program is selected based on validation set availability:

• With validation: The program with the best score is returned (with earlier programs winning ties)
• Without validation: The latest program is returned

If an optimization step fails:

• The error is logged with full traceback
• Optimization stops early
• The best program found so far is returned
• flag_compilation_error_occurred is set to True

The returned program includes two additional attributes:

• candidate_programs: List of all evaluated programs with their scores and strategies, sorted by score (best first). When an error occurs, this contains all successfully evaluated programs up to the point of failure.
• flag_compilation_error_occurred: Boolean indicating if any optimization step failed during compilation.

4. Further Details

Model Lifecycle Management: For local models (like LocalLM), BetterTogether automatically launches models before first use, kills them after optimization completes, and relaunches them after fine-tuning when model names change. These operations are no-ops for API-based LMs but needed for local model serving.

Custom Compile Arguments: You can pass custom compile arguments to specific optimizers using the optimizer_compile_args parameter:

• Override default arguments: Pass custom trainset/valset/teacher to specific optimizers
• Customize per optimizer: Each optimizer can have different compile arguments (e.g., num_trials, max_bootstrapped_demos)

Note: The student argument cannot be included in optimizer_compile_args - BetterTogether manages the student program for all optimizers. See the compile() method docstring for detailed argument documentation.

Best Practices

When to Use BetterTogether

BetterTogether is the right optimizer when:

• You want to squeeze every bit of performance: Prompt optimization is often the best bang for buck, quickly discovering high-level strategies. When opportunity allows, adding weight optimization on top compounds these gains, yielding benefits that exceed either approach alone.
• You have fine-tuning capabilities: Weight optimizers like BootstrapFinetune require LMs with a fine-tuning interface. Currently supported: LocalProvider, DatabricksProvider, and OpenAIProvider. You can extend the Provider class for custom use cases, or use BetterTogether to combine prompt optimizers only.

The Databricks case study demonstrates this effectiveness. They evaluated on IE Bench—a comprehensive suite spanning enterprise domains (finance, legal, commerce, healthcare) with complex challenges: 100+ page documents, 70+ extraction fields, and hierarchical schemas. Using GPT-4.1:

• SFT alone: +1.9 points over baseline
• GEPA alone: +2.1 points over baseline (slightly exceeding SFT)
• GEPA + SFT (BetterTogether): +4.8 points over baseline

This demonstrates that prompt optimization can match or surpass supervised fine-tuning, and combining these techniques yields strong compounding benefits.

Common Strategies and Optimizers

Common strategies:

• "p -> w": Optimize prompts first, then fine-tune (simple and often effective)
• "p -> w -> p": Optimize prompts, fine-tune, then optimize prompts again (can build on fine-tuning improvements)
• "w -> p": Fine-tune first, then optimize prompts

Example optimizer combinations:

• GEPA + BootstrapFinetune: Prompt optimization with fine-tuning
• MIPROv2 + BootstrapFinetune: Prompt optimization with fine-tuning
• Multiple prompt optimizers: Alternate between different prompt optimization approaches (experimental)

Further Reading

• BetterTogether Paper: arxiv:2407.10930
• Databricks Case Study - Real-world application combining BetterTogether with GEPA
• Optimizers: choosing one