Triage Lab - Experiment Records

This skill manages experiment records and optimization history for triage logic.

Current Implementation

File: java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java

Signal Priority (classifyPage method)

1. hasTableBorder - TableBorder presence (confidence: 1.0)
2. hasVectorTableSignal - Grid lines, border lines, line art (confidence: 0.95)
3. hasTextTablePattern - Text patterns with consecutive validation (confidence: 0.9)
4. hasSuspiciousPattern - Y-overlap or large gap detection (confidence: 0.85)
5. lineToTextRatio > 0.3 - High line chunk ratio (confidence: 0.8)
6. alignedLineGroups >= 5 - Aligned baseline groups (confidence: 0.7)

Key Thresholds

Experiment History

Experiment 001 (2026-01-03): FP Cause Analysis

Goal: Identify root causes of high False Positive rate

Baseline:

• Documents: 200 (42 with tables)
• TP: 41, TN: 48, FP: 110, FN: 1
• Precision: 27.15%, Recall: 97.62%, F1: 42.49%

FP by Signal:

Root Cause: Y-overlap check in hasSuspiciousPattern is too sensitive

• Condition previous.getTopY() < current.getBottomY() triggers on normal multi-column layouts

Experiments:

Recommendation:

• To maintain recall: Remove Y-overlap check (Precision +9%, Recall unchanged)
• To optimize F1: Use only reliable signals (F1 +22%, Recall -7%)

FN Documents:

• 01030000000110: Missed by all experiments (needs investigation)
• 01030000000122, 01030000000116, 01030000000117: Only detected by SuspiciousPattern

Applied: Y-overlap check removed (2026-01-03)

Experiment 002 (2026-01-03): Further FP Reduction

Goal: Reduce remaining 72 FPs after Y-overlap removal

Current FP by Signal (after Experiment 001):

Experiment 2A: Gap Multiplier (hasSuspiciousPattern)

→ No effect (Y-overlap removal already optimized this signal)

Experiment 2B: AlignedLineGroups Threshold

→ Recommended: Threshold 5 (FP -5, Recall maintained)

Experiment 2C: Vector Signal Criteria

→ Minimal effect (FP -1, higher values reduce Recall)

Recommendation:

• Apply alignedLineGroups threshold 3 → 5
• Expected: FP 64 → 59 (-5), Recall 92.86% (maintained), F1 +1.92%

Applied: alignedLineGroups threshold 3 → 5 (2026-01-03)

Actual Results:

Next Steps:

• Investigate hasTableBorder FPs (14 cases, external library)
• Investigate unknown FPs (8 cases)

Experiment 003 (2026-01-03): VectorTableSignal & SuspiciousPattern Analysis

Goal: Further reduce FP 67 while maintaining high Recall

Current FP by Signal (after Experiment 002):

VectorTableSignal Sub-signal Analysis (23 FPs):

→ hasAlignedShortLines is the primary cause of VectorSignal FPs

Experiments:

Analysis:

• hasTableBorder (14 FPs): External library, cannot modify
• hasVectorTableSignal (23 FPs): hasAlignedShortLines too aggressive
• hasSuspiciousPattern (19 FPs): Gap detection catches non-table layouts

Recommendation:

• Best for Recall: 003D (Disable SuspiciousPattern)
  • FP: 67 → 55 (-12), FN: 1 → 2 (+1), Recall: 95.24%
• Best for F1: 003I (Combined)
  • FP: 67 → 47 (-20), FN: 1 → 4 (+3), F1: 59.84%

FN Documents (003D):

• 01030000000122: Only detected by SuspiciousPattern (gap-based)
• 01030000000110: Never detected (needs separate investigation)

Applied: 003D - Disabled hasSuspiciousPattern (2026-01-03)

Actual Results:

Experiment 004 (2026-01-03): AlignedLineGroups Signal Analysis

Goal: Further reduce FP 55 while maintaining Recall 95.24%

Current FP by Signal (after Experiment 003):

VectorTableSignal Sub-signal Analysis (23 FPs):

Experiments:

Analysis:

• alignedLineGroups signal caused 12 FPs but detected no additional true tables
• Disabling it removes all 12 FPs without any FN increase
• Best option for maintaining Recall while improving Precision

Applied: 004D - Disabled alignedLineGroups signal (2026-01-03)

Actual Results:

Next Steps:

• Investigate hasVectorTableSignal FPs (23 remaining) - hasAlignedShortLines main cause
• Investigate hasTableBorder FPs (14 cases, external library limitation)

Experiment 005 (2026-01-03): Large Image Signal for FN Reduction

Goal: Reduce FN by detecting pages with large images (potential table/chart images)

Background:

• FN documents 01030000000110 and 01030000000122 contain images with tables
• 110: 28.64% page area (graph image)
• 122: 11.73% page area (table image)

Implementation:

• Added hasLargeImage signal to TriageProcessor
• Detects ImageChunk objects and calculates max image area / page area ratio

Experiments:

Analysis:

• 11% threshold achieves 100% Recall (all 42 table documents detected)
• Trade-off: FP increases from 43 → 83 (+40), Precision drops from 48% → 34%
• F1 decreases from 64% → 50% due to high FP increase
• Many FPs are documents with decorative images, diagrams, photos

Experiment 005B: Adding Aspect Ratio Condition

Observation: FN documents have wide images (ratio 1.79, 3.68), while FP documents often have square/tall images (ratio 0.6~1.5).

Final Configuration:

• Image area >= 11% of page area
• Image aspect ratio (width/height) >= 1.75

Trade-off:

• Achieves 100% Recall (all 42 table documents detected)
• FP increases from 43 → 55 (+12)
• F1 decreases from 64% → 60.43% (-3.57%)

Applied: 11% + aspect ratio 1.75 (2026-01-03)

Template for New Experiments

### Experiment XXX (YYYY-MM-DD): [Title]

**Goal**: [What are you trying to improve?]

**Changes**: [What did you modify?]

**Results**:
| Config | Precision | Recall | F1 | FP | FN |
|--------|-----------|--------|-----|-----|-----|
| Before | | | | | |
| After | | | | | |

**Conclusion**: [What did you learn? Should this be applied?]

**Next Steps**: [What to try next?]

How to Run Experiments

# Run triage accuracy test
./scripts/test-java.sh -Dtest=TriageProcessorIntegrationTest#testTriageAccuracyOnBenchmarkPDFs

# Debug specific document
./scripts/bench.sh --doc-id 01030000000110

Related Files

• TriageProcessor.java
• TriageProcessorIntegrationTest.java