evaluate-document-intelligence — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited evaluate-document-intelligence (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
| Metric | Target | How to Measure |
|---|---|---|
| Field extraction accuracy | ≥ 95% (prebuilt), ≥ 90% (custom) | Labeled test set comparison |
| Table recognition F1 | ≥ 90% | Cell-level precision/recall |
| Confidence calibration | Within ±5% of actual accuracy | Confidence vs actual accuracy |
| Processing time per page | < 10 seconds | Timed batch processing |
| Human review rate | < 15% | Fields below confidence threshold |
| PII detection recall | ≥ 99% | Known PII pattern test |
| End-to-end pipeline success | ≥ 98% | Documents processed without errors |
Create labeled test documents:
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics extraction_accuracypython evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics table_recognitionpython evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics performancepython evaluation/eval.py --all --output evaluation/report.json --ci-gate| Result | Action |
|---|---|
| All PASS | Deploy to production |
| Extraction < 85% | Retrain custom model with more samples |
| Table F1 < 80% | Switch to Layout model for complex tables |
| High human review rate (>25%) | Lower confidence threshold or retrain |
| Processing > 15s/page | Enable parallel processing, check resource SKU |
| PII recall < 95% | Block deployment, fix PII detection config |
| Symptom | Root Cause | Fix |
|---|---|---|
| All fields empty | Wrong model selected | Match model to document type |
| Numbers extracted as text | Locale mismatch | Set correct locale (en-US, de-DE) |
| Table columns misaligned | Complex multi-span headers | Use Layout model + custom post-processing |
| Handwriting not recognized | Prebuilt model limitation | Use custom model trained on handwriting |
| Date format wrong | Regional format difference | Normalize dates in post-processing |
| Low confidence everywhere | Poor scan quality | Require 300 DPI minimum in intake |
# Add quality gates to deployment pipeline
- name: Extraction Accuracy Gate
run: python evaluation/eval.py --metrics extraction_accuracy --ci-gate --threshold 0.90
- name: Table Recognition Gate
run: python evaluation/eval.py --metrics table_recognition --ci-gate --threshold 0.85
- name: PII Detection Gate
run: python evaluation/eval.py --metrics pii --ci-gate --threshold 0.99~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.