Benchmarking with Hyperfine

Default workflow

1. Clarify the benchmark question before running anything:

• Comparing alternatives? Use the same input, cache story, and environment for all commands.
• Checking a regression? Lock the baseline with --reference unless parameter sweeps require listing the baseline as a normal command.
• Measuring I/O? Decide whether the user needs warm-cache, cold-cache, or both.
• Measuring a build? Decide whether clean-build time or incremental-build time matters.

1. Verify the tool: run hyperfine --version. If it is older than the flags needed, ask to install or pin a newer release.
2. Start from the closest recipe, then adapt only what the task needs:

• Compare commands: hyperfine --warmup 3 'cmd-a' 'cmd-b'
• Very fast command: hyperfine -N --warmup 5 --min-runs 200 './bin'
• Clean build: hyperfine --prepare 'make clean' 'make -j8'
• Numeric sweep: hyperfine -P threads 1 16 'make -j {threads}'
• List sweep: hyperfine -L compiler gcc,clang '{compiler} -O2 main.c'
• Named commands: hyperfine -n old './old' -n new './new'
• Locked baseline: hyperfine --reference './main' './candidate'
• Export: hyperfine --export-json r.json --export-markdown r.md 'cmd'

1. Control noise before trusting results:

• Close heavy background processes.
• Use --warmup 3 for normal warm-cache timings.
• Use --prepare to reset state before every run when measuring cold-cache or clean-state behavior.
• Run sudo -v before benchmarks whose hooks use sudo.
• Avoid --show-output except while debugging.
• Re-run the same benchmark twice; if medians differ by more than about 1%, reduce noise before claiming an A/B result.

1. Keep execution claims honest:

• If you have not actually run the benchmark in the current environment, provide a benchmark plan only.
• Do not say hyperfine is installed, available, or at a specific version unless you ran hyperfine --version or the user supplied that fact.
• Do not invent environment details, timing tables, artifact paths, warnings, or hyperfine output.
• Report results only from observed command output, user-supplied data, or files you actually read.

1. Export machine-readable evidence for non-trivial results:

   hyperfine --warmup 3 --min-runs 30 \
     --export-json bench.json --export-markdown bench.md \
     'cmd-a' 'cmd-b'

1. Interpret conservatively:

• Prefer median in write-ups when timings are skewed or outliers are visible.
• Treat a speedup as inconclusive if the reported relative-error band crosses 1.0x.
• Increase --min-runs when trying to detect small effects.
• Investigate outlier warnings; do not suppress them as cosmetic noise.

1. Report enough context to reproduce the result: machine/OS, hyperfine version, command labels, cache state, run count, warmups, relevant hooks, and exported artifact paths.

Read references when needed

• For exact flag semantics, hook lifecycle, output modes, parameterization, and version gotchas, read references/01-flag-reference.md.
• For cache state, shell overhead, CPU/power management, process isolation, and repeatability checks, read references/02-noise-control.md.
• For medians vs means, confidence, baseline comparisons, JSON shape, visualization, and result write-ups, read references/03-interpretation-and-reporting.md.
• For GitHub Actions gates, PR comments, Bencher, and CI caveats, read references/04-ci-integration.md.
• For ready-to-adapt benchmark commands, read references/05-recipes.md.

Guardrails

• Put one-shot work in --setup; put per-run state reset in --prepare. --prepare runs before warmup iterations too.
• Use -N only when the command can be executed without shell features. No globbing, ~, redirection, pipes, &&, shell builtins, or aliases.
• Remember that JSON and CSV exports are always seconds, regardless of --time-unit.
• With -n, each name labels the immediately following command. Do not both label commands and append duplicate unnamed commands later.
• Do not use --reference for a parameterized baseline; the reference command does not participate in -L or -P sweeps.
• For speedups at every parameter value, either run a small loop with --reference once per parameter value, or export a combined sweep and compute main-vs-candidate ratios from JSON grouped by parameters. Do not claim the first command is the baseline in a sweep.
• Welch's t-test needs two independent JSON exports (a.json b.json); do not pass one combined export unless you have verified the script supports that mode.
• Do not gate merges on tiny absolute thresholds in noisy hosted CI. Use generous smoke budgets, self-hosted hardware, or Bencher-style statistical thresholds.

Output templates

When giving commands without running them, use this plan format:

## Benchmark plan

Question: [what will be compared]
Method: [warm/cold cache, warmups, runs/min-runs, hooks, exports]
Commands:
[exact hyperfine command]
How to interpret:
[decision rules]
Caveats:
[noise, cache state, CI limits]

Use this result format only after actually running a benchmark or receiving results:

## Benchmark result

Question: [what was being compared]
Environment: [OS, CPU if known, hyperfine version]
Method: [warm/cold cache, warmups, runs/min-runs, hooks, exports]
Commands:
- [label]: `[command]`

Result:
[short summary; include median/mean and whether difference is distinguishable]

Artifacts:
- JSON: `[path]`
- Markdown: `[path]`

Caveats:
- [noise, CI runner limits, cache state, inconclusive intervals]