QA-Eval: Quality Evaluation Framework

Run systematic quality evaluations on OCR and translation output across all scripts and languages in Source Library. Produces structured JSON results and markdown blog posts suitable for academic publication.

Issue: #1329

Quick Start

# Load env
set -a; source .env.production.local; set +a

# OCR consistency (run each model N times, compute Modal Consistency Rate)
node scripts/eval/qa-eval.mjs consistency --corpus=bhutan --sample=10 --models=flash,opus --runs=3

# Embedding-space evaluation (hallucination detection without ground truth)
node scripts/eval/qa-eval.mjs embedding --corpus=bhutan --sample=10

# Compare against ground truth (CER for OCR, BLEU/ROUGE for translation)
node scripts/eval/qa-eval.mjs compare --corpus=bhutan --against=ocr

# Readiness score for a corpus
node scripts/eval/qa-eval.mjs readiness bhutan

# Show all results
node scripts/eval/qa-eval.mjs report --latest

# Generate blog post from results
node scripts/eval/qa-eval.mjs report --corpus=bhutan --format=blog --save

Invocation Modes

Interactive

/qa-eval                                    # Show help and available corpora
/qa-eval --corpus=bhutan --sample=5         # Quick consistency check
/qa-eval --corpus=bhutan --blog             # Full eval + blog post

Specific Commands

/qa-eval consistency --corpus=bhutan --models=flash,opus --runs=3
/qa-eval embedding --corpus=bhutan --sample=20
/qa-eval compare --corpus=bhutan --against=translation
/qa-eval matrix                             # All corpora comparison table
/qa-eval readiness bhutan                   # Quick readiness score

Cost Estimation

/qa-eval consistency --corpus=bhutan --sample=10 --models=flash,opus --runs=3 --dry-run

Available Corpora

Defined in scripts/eval/corpus-registry.json:

Model Aliases

Metrics

OCR Quality

• MCR (Modal Consistency Rate): % of N runs producing the majority output at temp=0
• Pairwise character similarity: Levenshtein-based, 0-100%
• Syllable similarity: Script-aware tokenization (tsheg for Tibetan, char for CJK)
• CER (Character Error Rate): Edit distance / reference length (requires ground truth)

Translation Quality

• BLEU-4: N-gram overlap with brevity penalty (requires ground truth)
• ROUGE-L: Longest common subsequence F1 (requires ground truth)
• Embedding distance: Cosine distance between OCR and translation embeddings (no ground truth needed)

Hallucination Detection

• Pages where OCR→Translation embedding distance exceeds 2σ from corpus mean are flagged
• Example: Flash Lite "translating" an astrological text as a ritual manual

Readiness Score

• High: MCR ≥ 90% AND cross-model agreement ≥ 85%
• Medium: MCR ≥ 70% AND cross-model agreement ≥ 70%
• Low: Below medium thresholds

Output

Results are saved to scripts/eval/results/ as JSON and optionally as markdown blog posts in docs/.

scripts/eval/results/
  bhutan-consistency-2026-04-23.json
  bhutan-embedding-2026-04-23.json
  matrix-2026-04-23.json
docs/
  qa-eval-bhutan-2026-04-23.md

Ground Truth

Place reference transcriptions and translations in scripts/eval/ground-truth/ as JSON:

{
  "book_id": "abc123",
  "page_number": 5,
  "script": "tibetan",
  "source": "BDRC etext",
  "source_url": "https://library.bdrc.io/...",
  "ocr_ground_truth": "...",
  "translation_ground_truth": "...",
  "translation_source": "Thurman 1994"
}

Sources: BDRC etexts, OpenPecha, Esukhia Derge Kangyur, Lotsawa House, scholarly editions.

Architecture

scripts/eval/
  qa-eval.mjs              # CLI entrypoint
  lib/
    metrics.mjs            # All metric functions
    runners.mjs            # Gemini + Claude model execution
    sampling.mjs           # MongoDB page sampling
    report.mjs             # JSON + Markdown output
    embedding-eval.mjs     # Embedding-space evaluation
  corpus-registry.json     # Known corpora
  ground-truth/            # Reference data
  results/                 # Output

Key References

• Blog post: docs/blog-tibetan-ocr-benchmark.md
• Prototype: _tmp-ocr-consistency.mjs
• Embedding model: gemini-embedding-2-preview (768d, matches production search)
• Related papers: GlotOCR Bench, Wang & Wang 2025, Conformal Risk Control for OCR