Evaluate Fine-Tuning Workflow

When to Use

• Compare fine-tuned model against base model
• Check for overfitting (training vs validation loss)
• Measure task-specific improvement metrics
• Validate model quality before deployment
• Gate promotion from staging to production

Evaluation Metrics & Targets

Step 1: Analyze Training Curves

python evaluation/analyze_training.py --job-id $JOB_ID

• Plot training loss vs validation loss over epochs
• Healthy: Both decreasing, gap < 0.1
• Overfitting: Training loss decreasing, validation increasing → reduce epochs
• Underfitting: Both high → increase epochs or learning rate

Step 2: Prepare Evaluation Test Set

Create test samples NOT in training data:

{"messages": [{"role": "user", "content": "What is the return window?"}], "expected": "30 days from purchase date", "category": "policy"}
{"messages": [{"role": "user", "content": "Can I get a refund for digital items?"}], "expected": "Digital items are non-refundable", "category": "policy"}

Minimum: 50 test samples, diverse across categories.

Step 3: Side-by-Side Comparison

python evaluation/eval.py --test-set evaluation/test-set.jsonl \
  --models base:gpt-4o-mini ft:$FINE_TUNED_MODEL \
  --metrics accuracy,format_compliance,hallucination

• Run identical prompts through both models
• Compare: accuracy, format adherence, response quality
• Score each response on 1-5 scale (automated + human spot-check)

Step 4: Domain-Specific Evaluation

Depending on fine-tuning use case:

Step 5: Check for Regression

• Run base model's original benchmark suite on fine-tuned model
• Fine-tuned model should NOT regress on general capabilities
• If regression > 5%: training data may be too narrow (catastrophic forgetting)

Step 6: Generate Evaluation Report

python evaluation/eval.py --full-report --output evaluation/report.json

Promotion Gate Decision

Common Evaluation Pitfalls

Evaluation Cadence

• Post-training: Full evaluation before any deployment
• Pre-promotion: Side-by-side comparison at each stage (dev→staging→prod)
• Monthly: Re-evaluate with fresh test data on deployed model
• On data update: Re-evaluate if training data is augmented
• On base model update: Re-evaluate if Azure updates the base model version