SOTA Performance Engineering

Purpose

Make systems fast by default and find why they are slow by evidence. This skill encodes two disciplines that share one rule set:

1. BUILD — write code whose performance characteristics are known, budgeted,

and protected by regression tests before it ships.

1. AUDIT — read existing code and telemetry to locate bottlenecks, rank them

by user-facing impact, and prescribe fixes with expected gains.

Core doctrine: measure first, but fix known pathologies on sight. Profiling is mandatory before micro-optimization; it is NOT required to remove an O(n²) loop, an N+1 query, or an unbounded cache. "Premature optimization" never excuses shipping a known pathology.

BUILD mode

When writing new code or features:

1. Set a budget before writing. Define the latency budget (p99, not average)

and decompose it across hops. An endpoint with a 200 ms p99 budget that calls auth (10 ms) + 2 DB queries (2×15 ms) + serialization (5 ms) has 155 ms of headroom — spend it consciously. See rules/01-methodology.md.

1. Choose data structures by access pattern, not habit. Know the n. n < 100:

anything works. n unbounded: complexity class is the design. See rules/02-algorithms-data-structures.md.

1. Batch and stream at every boundary. One round trip per collection, not

per item. Stream large results; never materialize unbounded data.

1. Control allocation in hot paths. Pre-size collections, reuse buffers,

avoid per-iteration allocation in loops that run > 10⁴ times per second. See rules/03-memory.md.

1. Make I/O cheap by construction. Pooled connections, keep-alive, buffered

writes, compression chosen per payload type. See rules/04-io-network.md.

1. Cache deliberately or not at all. Every cache ships with: key schema,

TTL + jitter, eviction policy, invalidation path, stampede protection, and a hit-ratio metric. A cache missing any of these is a future incident. See rules/05-caching.md.

1. Protect the win. Add a benchmark or perf test in CI for any code with a

budget. A perf improvement without a regression gate is a loan, not an asset.

1. Frontend ships against Core Web Vitals budgets (LCP ≤ 2.5 s, INP ≤ 200 ms,

CLS ≤ 0.1 at p75). See rules/06-frontend-web.md.

AUDIT mode

How to find performance issues by reading code

Work outside-in, hottest path first:

1. Identify the hot paths. Entry points with highest traffic or strictest

SLO: request handlers, queue consumers, render loops, cron jobs over large datasets. Audit those first; ignore cold admin paths until the end.

1. Grep for pathology signatures (high hit rate, low effort):

• Loops containing await/network/DB calls → N+1 (rules/02)
• String/array concatenation inside loops → accidental quadratic (rules/02)
• .includes/in list/linear find inside a loop → O(n·m) (rules/02)
• SELECT *, queries without LIMIT, missing pagination (rules/02, rules/04)
• Caches/maps with insert but no eviction or TTL → leak (rules/03, rules/05)
• addEventListener/subscribe without matching removal (rules/03)
• Sequential awaits on independent operations → serialized latency (rules/04)
• New client/connection per request instead of pooled (rules/04)
• Sync file/crypto/compression calls on async event loops (rules/04)
• JSON.parse/serialize of large payloads in hot loops (rules/03)

1. Check the boundaries. Most production latency lives at boundaries:

process↔kernel (syscalls), service↔DB, service↔service, server↔browser. Count round trips per user action; > 3 sequential round trips is a finding.

1. Check resource lifecycle. Anything created per-request that is expensive

to create (connections, TLS sessions, regexes, compiled templates, clients) should be created once and reused.

1. Check what's missing: no timeouts, no pagination, no backpressure, no

pool bounds, no cache eviction — absent code is the most common perf bug.

What to measure (when you can run the system)

• Latency distribution: p50/p95/p99 per endpoint — never averages

(rules/01). Compare p99 to p50; ratio > 10× means contention, GC, or stampedes, not slow code.

• USE per resource (Utilization, Saturation, Errors): CPU, memory, disk,

network, pools, queues. RED per service (Rate, Errors, Duration).

• Where time goes: CPU flamegraph for compute, off-CPU/wall profile for

waiting. A request that is slow with idle CPU is blocked on I/O or locks.

• Allocation rate and GC pause time for managed runtimes.
• Cache hit ratios and DB round trips per request.
• Frontend: field CWV (CrUX/RUM) at p75, not lab-only Lighthouse.

Severity conventions (by user-facing impact)

Escalate one level if the code path is on the critical user journey (checkout, login, search) or if growth is super-linear with data/users.

Finding format

[SEVERITY] <one-line title>
Location: <file:line(s)>
Pattern: <pathology name, e.g. "N+1 query", "unbounded cache">
Evidence: <code excerpt or metric>
Impact: <quantified or estimated user-facing effect, with the math>
Fix: <specific change, with expected gain>
Verify: <how to confirm the fix: benchmark, profile, metric to watch>

Estimate impact with arithmetic, not adjectives: "200 items × 1 query × ~1 ms RTT = ~200 ms added per page view" beats "this is slow".

Rules index

Top-10 non-negotiables

1. Measure before optimizing; fix pathologies on sight. Profile before

micro-tuning. But N+1, O(n²) on unbounded input, unbounded caches, and sync-blocking the event loop need no profiler — fix them when you see them.

1. Percentiles, never averages. Report and budget p50/p95/p99. An average

hides the 1% of users who hit every cache miss and GC pause.

1. No I/O inside a loop over a collection. Batch it, join it, or

parallelize it with a bound. One round trip per item is always a finding.

1. **Every cache has bounded size, TTL with jitter, an invalidation path, and

stampede protection.** Otherwise it's a memory leak with a hit ratio.

1. Pool expensive resources. Connections, TLS sessions, threads, compiled

regexes, HTTP clients: create once, reuse always, bound the pool.

1. Stream unbounded data; never load "all rows" into memory. Paginate with

cursors, process in chunks, set LIMITs.

1. Never block an async event loop with sync file I/O, crypto, compression,

or CPU-heavy work. Offload to workers or use async variants.

1. Set timeouts and bounds on everything: requests, queries, pools, queues,

retries (with backoff + jitter). Missing bounds turn slowness into outage.

1. Sequential awaits on independent work are stolen latency. Run

independent I/O concurrently; the latency of the batch is the max, not sum.

1. Protect every win with a regression gate. A benchmark in CI with

variance-aware thresholds, or the regression returns within a quarter.