OCR Accuracy Benchmark

Evaluates deepseek-ai/DeepSeek-OCR-2 (and any compatible OCR VLM) on olmOCR-bench (AllenAI), the benchmark explicitly used in DeepSeek-OCR-2 official evaluations.

Targets olmOCR-bench because:

• Public HuggingFace dataset with 7,010 deterministic unit tests
• Explicitly cited by DeepSeek-OCR-2 authors
• Clear pass/fail semantics — no heavy CDM/TEDS/LaTeXML dependencies
• Covers 7 challenging document types across 1,403 PDF pages

Setup

# Step 0 (one-time): download olmOCR-bench including PDFs (~2 GB via Git LFS)
pip install huggingface_hub
hf download --repo-type dataset \
    allenai/olmOCR-bench --local-dir ./olmOCR-bench
# This places bench_data/  (7 JSONL files + pdfs/ directory) under ./olmOCR-bench/

# Required: benchmark dependencies (pymupdf is in sglang[test]; aiohttp/tqdm are in core)
pip install "sglang[test]"
# OR install PDF rendering manually (choose one):
#   pip install pymupdf          # recommended (faster, pure Python wheel)
#   pip install pdf2image        # needs poppler: sudo apt install poppler-utils

# Start the sglang server (matches run.sh in this repo)
python -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-OCR-2 \
    --host 127.0.0.1 --port 30000

Why the download step? The olmOCR-bench PDF files are stored in Git LFS on HuggingFace. datasets.load_dataset() cannot retrieve LFS-backed binary files, so the benchmark reads the JSONL test files and PDFs directly from a local clone of the repository.

Usage

# Full benchmark — all 7 splits (~7,010 tests)
python -m benchmark.ocr.bench_sglang \
    --port 30000 \
    --model deepseek-ai/DeepSeek-OCR-2 \
    --split all \
    --concurrency 8 \
    --output-dir ./ocr_bench_results

# Single split
python -m benchmark.ocr.bench_sglang --port 30000 --split arxiv_math --concurrency 16

# Quick smoke-test (50 samples from one split)
python -m benchmark.ocr.bench_sglang --port 30000 --split old_scans --max-samples 50

# Use "Free OCR" prompt instead of markdown conversion
python -m benchmark.ocr.bench_sglang --port 30000 --split all --prompt-mode free_ocr

# Save raw model outputs for inspection
python -m benchmark.ocr.bench_sglang --port 30000 --split multi_column --save-raw-outputs

Arguments

Test Classes (olmOCR-bench)

Note on math: The official olmOCR-bench uses KaTeX rendering + Playwright for bounding-box symbol matching. This benchmark uses a symbol-token proxy (no browser dependency). Scores on arxiv_math and old_scans_math may therefore differ from the official leaderboard.

Dataset Splits

Reference Scores

Column order matches the olmOCR README: AR = arxiv_math, OSM = old_scans_math, TA = table_tests, OS = old_scans, HF = headers_footers, MC = multi_column, LTT = long_tiny_text, Base = baseline.

* = scores reported by model authors, not reproduced by olmOCR team.

DeepSeek-OCR-2 per-split scores for OS/HF/MC/LTT are not officially reported; only the three highlighted splits and overall appear on the HuggingFace model card.

Note on math scores: This benchmark uses token-overlap matching (≥70% threshold) rather than the official KaTeX rendering + Playwright bounding-box comparison. Scores on arxiv_math and old_scans_math will therefore differ from the official leaderboard.

Sources: olmOCR README, DeepSeek-OCR-2 HF card.

Output Files

Results are written to --output-dir:

ocr_bench_results/
├── arxiv_math.json       # per-split detailed results
├── old_scans.json
├── ...
└── summary.json          # aggregated across all evaluated splits

Each split JSON contains:

• overall_score: % tests passed
• by_type: per-test-type pass rate
• total_tests, total_passed, error_samples
• Per-sample test_results with type, passed, optional error

Files