Postmortem Writing

Use this skill when authoring or reviewing an incident report (IR) for the Lorekeeper project. Every incident must produce durable process improvements — not just a fix but encoding the lesson into review patterns and checklists.

When to Use

• A production incident occurs (P0/P1)
• A CI pipeline breaks on main
• A bug with systemic root cause (not a one-off typo)
• The fix changes architecture patterns, not just code

Mandatory Deliverables

Every IR MUST deliver all of:

IR Template

YAML Frontmatter

---
**Status:** [Resolved | In Progress | Monitoring]
**Severity:** [P0/P1/P2]
**Date:** YYYY-MM-DD
**Author:** [Name]
**Fix PR:** #[number]
---

Sections

#### 1. Summary

One paragraph: what happened, why it was bad, what fixed it. Include the key insight in the first sentence.

#### 2. Timeline

Table of events in chronological order: time + event. Include first symptom, investigation start, fix identified, fix merged, recovery confirmed.

#### 3. Root Causes — Causal Chain

This is the most important section. Model the incident as a chain of independent links, where each link is survivable alone but together they create the failure.

Format each link as:

### 🥇 Link N: [Title]

[Technical explanation — what the link is]

**Why this caused today's crash:** [How this specific link contributed]

**Evidence:** [Log lines, metrics, code diffs, process output]

After the chain, show how the fixes break specific links:

| Fix                     | Which link it breaks | Why it suffices alone |
| ----------------------- | -------------------- | --------------------- |
| `busy_timeout=5000`     | Link 2               | ...                   |

#### 4. Impact

• Duration: [start to end — how long users were affected]
• Scope: [who/what was affected]
• Data integrity: [any data loss?]
• User-facing: [what users experienced]

#### 5. Detection

How the incident was first noticed (user report, alert, dashboard). Include evidence commands (e.g. ps aux, curl checks).

#### 6. Fixes Applied

Group by PR. For each PR, list changes as a table: what changed, file(s), what it does.

Separate emergency fixes (stop the bleeding) from structural fixes (eliminate the class).

#### 7. Action Items

Table: # | Action | Owner | Status

Every action item must be concrete and assigned. No "investigate further" — each item has a clear done condition.

Must include:

• Skill patches (lorekeeper-code-reviewer, github-code-review-checklist)
• Engineering docs if new pattern
• Any deferred items with explicit reason

#### 8. Lessons Learned

Numbered list of principles extracted from this incident. These go into the causal chain analysis section and become future review heuristics.

#### 9. Related

Links to: PRs, commits, related skills, engineering docs, previous IRs.

Causal Chain Technique

The causal chain is the key insight from IR-002. Rules:

1. Each link must be independently survivable — if link 1 existed alone, no incident.
2. Number links by causal proximity — 🥇 is the first domino, 🥈 is second, etc.
3. Provide evidence for each link — logs, code, process output.
4. End with a fix/chain-break table — prove each fix independently breaks the chain.
5. Include the dormant link — the bug that didn't cause this incident but would cause the next one (e.g. a missing commit() that passed all tests).

Chain diagram

Optionally include an ASCII causal chain diagram:

[Link 1 crash] ─→ [Link 2 instant error]
         │                │
         │                └──→ [User-visible symptom]
         │
         └──→ [Link 3 infinite retries] ─→ [Link 4 zombies amplify]
                                                  │
                                                  └──→ [Every init hits contention]

Post-Incident Checklist

After the IR is written, verify:

• [ ] IR filed in docs/incidents/ with sequential numbering
• [ ] BLOCKER patterns added to lorekeeper-code-reviewer (with concrete code examples)
• [ ] Checklist items added to github-code-review-checklist
• [ ] Engineering docs created/updated in docs/engineering/
• [ ] Action items all have owners and statuses
• [ ] Fix PRs referenced by number
• [ ] Previous IRs referenced in Related section
• [ ] Version bumped on any changed skills