ContextQMD
Back to Ruflo

SWE-Bench Official Implementation Complete

v2/benchmark/swe-bench-official/IMPLEMENTATION_COMPLETE.md

3.6.304.1 KB
Original Source

SWE-Bench Official Implementation Complete

✅ Implementation Status

We have successfully integrated the official SWE-bench evaluation system with the swarm-bench CLI tool. The implementation is complete and ready for full evaluation runs.

🏗️ What Was Built

1. Official SWE-bench Integration (official_integration.py)

  • Connects to HuggingFace datasets API
  • Downloads official SWE-bench and SWE-bench-Lite datasets
  • Executes claude-flow commands with proper formatting
  • Generates predictions.json for leaderboard submission
  • Validates submission format

2. CLI Integration

  • swarm-bench swe-bench official - Run official evaluation
  • swarm-bench swe-bench official --limit 10 - Run limited test
  • swarm-bench swe-bench official --validate - Validate predictions

3. Command Format

The system uses the correct claude-flow command format:

bash
npx claude-flow@alpha swarm "task" --strategy optimization --agents 8 --non-interactive
npx claude-flow@alpha hive-mind spawn "task" --agents 8 --non-interactive

4. Optimal Configuration

Based on our testing, the optimal configuration is:

  • Mode: mesh (for complex coordination)
  • Strategy: optimization (for bug fixes)
  • Agents: 8 (balanced parallelism)

📊 Performance Expectations

Timing

  • Each SWE-bench instance: 5-10 minutes
  • SWE-bench-Lite (300 instances): ~25-50 hours
  • Full SWE-bench (2,294 instances): ~200-400 hours

Resources

  • Requires active claude-flow with API access
  • Each instance makes multiple API calls
  • Generates git diff patches for each issue

🚀 How to Run

Quick Test (1 instance)

bash
python run_swe_bench_optimized.py --quick

Limited Evaluation (5 instances, ~30 min)

bash
python run_swe_bench_optimized.py --evaluate

Full SWE-bench-Lite (300 instances, ~25 hours)

bash
swarm-bench swe-bench official

With Progress Tracking

bash
swarm-bench swe-bench official --limit 10 | tee swe-bench.log

📁 Output Files

The system generates:

  • predictions.json - Submission file for leaderboard
  • evaluation_report_*.json - Detailed metrics
  • Individual patch files for each instance

⚠️ Important Notes

  1. Long Runtime: Full evaluation takes many hours
  2. API Usage: Each instance uses multiple API calls
  3. Timeout: Default 10 minutes per instance
  4. Validation: Always validate before submission

🎯 Submission Process

  1. Run evaluation:

    bash
    swarm-bench swe-bench official --limit 300

  2. Validate predictions:

    bash
    swarm-bench swe-bench official --validate

  3. Submit to leaderboard:

📈 Optimization Results

The system has been optimized with:

  • Parallel agent execution (8 agents)
  • Mesh coordination for complex tasks
  • Optimization strategy for bug fixes
  • Proper timeout handling
  • Efficient prompt engineering

🔧 Troubleshooting

If patches aren't generating:

  1. Verify claude-flow is installed: npx claude-flow@alpha --version
  2. Check API access is configured
  3. Try increasing timeout in config
  4. Run with --limit 1 to test single instance

📝 Code Structure

benchmark/
├── src/swarm_benchmark/swe_bench/
│   ├── official_integration.py  # Main integration
│   ├── engine.py                # Original engine
│   ├── datasets.py              # Dataset handling
│   ├── evaluator.py             # Evaluation logic
│   ├── metrics.py               # Performance tracking
│   └── optimizer.py             # Configuration optimization
├── run_swe_bench_optimized.py   # Optimization runner
├── run_real_swe_bench.py        # Simple runner
└── swe-bench-official/          # Results directory

✅ Ready for Production

The implementation is complete and ready for:

  • Official SWE-bench evaluation
  • Leaderboard submission
  • Performance benchmarking
  • Configuration optimization

The system correctly uses claude-flow commands and generates predictions in the required format for official submission.