Tune Document Intelligence

When to Use

• Optimize field extraction accuracy for specific document types
• Tune confidence thresholds to balance automation vs human review
• Optimize processing throughput for batch operations
• Configure model routing for cost optimization
• Improve custom model performance with retraining

Tuning Dimensions

Dimension 1: Model Selection Optimization

Decision tree:

1. Does a prebuilt model exist for this doc type? → Use prebuilt (cheapest, most accurate)
2. Is layout consistent across documents? → Train custom model
3. Is layout variable? → Use Layout + GPT-4o post-processing

Dimension 2: Confidence Threshold Tuning

Diagnostic: Run python evaluation/eval.py --metrics confidence_calibration

Dimension 3: Processing Throughput

Optimization steps:

1. Split multi-page PDFs into single pages for parallel processing
2. Use Azure Storage queue for async processing
3. Set appropriate SKU tier (S0 for production throughput)
4. Implement retry with exponential backoff for throttling

Dimension 4: Cost Per Document

Monthly cost estimate (10,000 documents/day, avg 3 pages):

• Document Intelligence: ~$9,000/mo (900K pages × $0.01)
• Storage: ~$200/mo (hot + cool tiers)
• GPT-4o enrichment (10% of docs): ~$500/mo
• Total: ~$9,700/mo (optimize to ~$6,000 with page skipping + model routing)

Dimension 5: Custom Model Retraining

When to retrain:

• Accuracy drops below 88% on new documents
• New document layouts introduced
• New fields need to be extracted
• Customer feedback indicates systematic errors

Retraining checklist:

1. Collect 10-20 new samples of failing document types
2. Label in Document Intelligence Studio
3. Retrain with combined old + new samples
4. Compare new model accuracy vs previous version
5. Deploy only if new model is better on both old and new samples

Production Readiness Checklist

• [ ] Extraction accuracy ≥ 90% across all document types
• [ ] Confidence threshold calibrated per document type
• [ ] Human review rate < 15%
• [ ] Processing throughput meets batch requirements
• [ ] PII detection and redaction verified
• [ ] Error handling and retry logic tested
• [ ] Cost per document within budget
• [ ] Custom model versioned and rollback-capable
• [ ] Monitoring dashboards showing extraction metrics

Output: Tuning Report

After tuning, generate comparison:

• Extraction accuracy delta per field
• Human review rate change
• Cost per document change
• Processing throughput improvement
• Confidence threshold recommendations per document type