evaluate-edge-ai-phi4 — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited evaluate-edge-ai-phi4 (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
| Metric | Target | How to Measure |
|---|---|---|
| Inference latency (TTFT) | < 2s on edge, < 500ms on laptop | Time from prompt to first token |
| Tokens per second | ≥ 5 tok/s edge, ≥ 20 tok/s laptop | Streaming output timing |
| Peak memory usage | < device RAM × 80% | Process memory monitoring |
| Model load time | < 10s edge, < 3s laptop | Cold start timing |
| Quality vs cloud | ≥ 85% of GPT-4o quality | Side-by-side eval on test set |
| Offline success rate | 100% | Inference with network disabled |
| Battery impact | < 5% per 100 inferences | Battery level monitoring |
| Storage footprint | < 50% of device storage | Model + runtime size |
python evaluation/benchmark_edge.py --model models/phi4-int4/ \
--prompts evaluation/test-set.jsonl --warmup 3 --iterations 50Measure per device type:
python evaluation/eval.py --test-set evaluation/test-set.jsonl \
--models edge:models/phi4-int4/ cloud:gpt-4o \
--metrics accuracy,coherence,completeness| Test | FP16 Score | INT8 Score | INT4 Score |
|---|---|---|---|
| Classification | Baseline | Run test | Run test |
| Summarization | Baseline | Run test | Run test |
| Q&A | Baseline | Run test | Run test |
| Code generation | Baseline | Run test | Run test |
If any category drops > 5% from FP16 → use less aggressive quantization.
python evaluation/benchmark_edge.py --full-report --output evaluation/edge-report.json| Result | Action |
|---|---|
| All PASS | Deploy to edge fleet |
| Latency > 5s | Use more aggressive quantization or smaller model |
| Quality < 80% of cloud | Specific task not suitable for edge — keep in cloud |
| OOM on device | Reduce model size, check other processes |
| Battery drain > 10%/100inf | Optimize batch size, reduce inference frequency |
| Symptom | Root Cause | Fix |
|---|---|---|
| Very slow first inference | Cold model load | Pre-load model at device boot |
| Output quality poor | Over-quantized | Switch INT4 → INT8 |
| Device runs hot | CPU at 100% sustained | Limit threads, add inference cooldown |
| Inconsistent results | Non-deterministic without seed | Set seed in ONNX Runtime session |
| Crashes after 100+ queries | Memory leak | Check session cleanup, use session pool |
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.