Tune Call Center Voice AI (Play 04)

Optimize voice pipeline for latency, accuracy, and escalation quality.

When to Use

• End-to-end voice latency exceeds 2s target
• STT transcription accuracy below 95%
• TTS voice sounds unnatural
• Escalation triggers missing or false-firing

Latency Budget

STT Accuracy Tuning

# Test transcription accuracy
def eval_stt_accuracy(test_audio: list[dict]) -> dict:
    correct = 0
    for sample in test_audio:
        transcript = transcribe(sample["audio_path"])
        if normalize(transcript) == normalize(sample["expected_text"]):
            correct += 1
    return {"accuracy": correct / len(test_audio), "n": len(test_audio)}

# Tuning levers:
# - Language: set speech_recognition_language precisely
# - Noise: enable noise suppression
# - Custom model: train on domain-specific vocabulary

TTS Voice Selection

speech_config.speech_synthesis_voice_name = "en-US-JennyNeural"

Escalation Tuning

ESCALATION_TRIGGERS = [
    "speak to a human", "transfer me", "supervisor",
    "this is urgent", "I want to complain", "real person",
    "you're not helping", "I need help",
]

# Tune: Add/remove phrases based on false positive/negative analysis
def eval_escalation(test_transcripts: list[dict]) -> dict:
    tp, fp, fn = 0, 0, 0
    for t in test_transcripts:
        predicted = should_escalate(t["transcript"])
        if predicted and t["should_escalate"]: tp += 1
        elif predicted and not t["should_escalate"]: fp += 1
        elif not predicted and t["should_escalate"]: fn += 1
    precision = tp / (tp + fp) if (tp + fp) else 0
    recall = tp / (tp + fn) if (tp + fn) else 0
    return {"precision": precision, "recall": recall}

LLM Response Tuning

{
  "model": "gpt-4o-mini",
  "temperature": 0.3,
  "max_tokens": 150,
  "system_prompt": "You are a helpful call center agent. Be concise — this is voice, not text. Keep responses under 3 sentences."
}

Troubleshooting