Incident Response

Structured workflow for diagnosing and resolving production incidents.

When to Use This Skill

Use this skill when:

• An alert has fired
• Users report issues
• Monitoring shows anomalies
• System behavior is unexpected
• You're the on-call responder

Incident Response Phases

Phase 1: Triage (First 5 minutes)

Goal: Assess severity and impact.

#### Quick Assessment Questions

1. What is broken? (service, feature, infrastructure)
2. Who is affected? (all users, subset, internal only)
3. When did it start? (correlate with deployments/changes)
4. Is it getting worse? (check error rate trend)

#### Severity Classification

Phase 2: Context Gathering (5-15 minutes)

Goal: Collect data to understand the problem.

#### Kubernetes Context

# Cluster health overview
kubectl get nodes
kubectl get pods -A | grep -v Running | grep -v Completed

# Recent events
kubectl get events -A --sort-by='.lastTimestamp' | tail -30

# Check specific service
kubectl get pods -l app=<service-name> -o wide
kubectl logs -l app=<service-name> --tail=100

#### Recent Changes

# Recent deployments
kubectl get deployments -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,NAME:.metadata.name,UPDATED:.metadata.creationTimestamp' \
| sort -k3 -r | head -10

# Git history (if in repo)
git log --oneline --since="2 hours ago"

#### Metrics Check

# If Prometheus available
# Check error rates, latency, traffic

# If CloudWatch
aws cloudwatch get-metric-statistics \
  --namespace <namespace> \
  --metric-name <metric> \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 60 \
  --statistics Average

Phase 3: Diagnosis (15-30 minutes)

Goal: Identify root cause.

#### Common Root Causes

1. Recent deployment - Check if timing correlates
2. Resource exhaustion - CPU, memory, disk, connections
3. External dependency failure - Database, API, DNS
4. Configuration change - ConfigMaps, secrets, feature flags
5. Traffic spike - Unexpected load
6. Certificate/credential expiry - Auth failures
7. Infrastructure issue - Node failure, network partition

#### Diagnosis Decision Tree

Is there a recent deployment?
├── Yes → Check deployment logs, consider rollback
└── No → Continue

Are pods crashing?
├── Yes → Check logs: kubectl logs <pod> --previous
└── No → Continue

Are resources exhausted?
├── Yes → Scale up or optimize
└── No → Continue

Is an external dependency failing?
├── Yes → Check dependency status, implement fallback
└── No → Continue

Is there a traffic spike?
├── Yes → Scale up, enable rate limiting
└── No → Escalate for deeper investigation

Phase 4: Mitigation (ASAP)

Goal: Restore service, even if root cause unknown.

#### Quick Mitigations

Rollback Deployment:

# Kubernetes rollback
kubectl rollout undo deployment/<name> -n <namespace>
kubectl rollout status deployment/<name> -n <namespace>

Scale Up:

# Increase replicas
kubectl scale deployment/<name> --replicas=5 -n <namespace>

Restart Pods:

# Rolling restart
kubectl rollout restart deployment/<name> -n <namespace>

Toggle Feature Flag:

# If feature flags available, disable problematic feature

Redirect Traffic:

# If multiple regions, redirect away from affected region

Phase 5: Communication

Goal: Keep stakeholders informed.

#### Status Update Template

**Incident Update - [SERVICE] - [SEV LEVEL]**

**Status:** Investigating / Identified / Mitigating / Resolved
**Impact:** [Who/what is affected]
**Start Time:** [When it started]
**Current Actions:** [What we're doing]
**Next Update:** [When to expect next update]

---
Incident Commander: [Name]

#### Communication Cadence

Phase 6: Resolution

Goal: Confirm service is restored.

#### Verification Checklist

• [ ] Error rates returned to baseline
• [ ] Latency returned to baseline
• [ ] All pods healthy
• [ ] Synthetic monitors passing
• [ ] User reports have stopped

#### Close Incident

**Incident Resolved - [SERVICE]**

**Duration:** [Start] to [End] ([X] minutes)
**Root Cause:** [Brief description]
**Resolution:** [What fixed it]
**Follow-ups:** [Links to action items]

Post-mortem scheduled: [Date/Time]

Post-Incident

Post-Mortem Template

# Incident Post-Mortem: [Title]

**Date:** [Date]
**Duration:** [Duration]
**Severity:** [SEV Level]
**Authors:** [Names]

## Summary
[2-3 sentence summary]

## Impact
- Users affected: [Number/percentage]
- Revenue impact: [If applicable]
- SLA impact: [If applicable]

## Timeline
- HH:MM - [Event]
- HH:MM - [Event]
- HH:MM - [Event]

## Root Cause
[Detailed explanation]

## Resolution
[How it was fixed]

## Lessons Learned
### What went well
- [Item]

### What could be improved
- [Item]

## Action Items
- [ ] [Action] - Owner: [Name] - Due: [Date]
- [ ] [Action] - Owner: [Name] - Due: [Date]

Quick Reference

Essential Commands

# Quick cluster health
kubectl get nodes && kubectl get pods -A | grep -v Running

# Service status
kubectl get pods,svc,endpoints -l app=<service>

# Recent events
kubectl get events --sort-by='.lastTimestamp' | tail -20

# Quick logs
kubectl logs -l app=<service> --tail=50 --all-containers

Escalation Contacts

Document your escalation path:

1. Primary on-call
2. Secondary on-call
3. Team lead
4. Engineering manager
5. VP Engineering (SEV1 only)

Related Skills

• k8s-debug: For Kubernetes-specific debugging
• log-analysis: For log pattern analysis
• argocd-gitops: For GitOps rollbacks