name: property-based-testing
# description: routing-facing summary of when this skill activates and what it covers.
description: "Use when reasoning about tests that specify universal properties of code rather than specific input-output pairs: the forall(input) → property quantification, the generator/shrinker primitives that produce inputs and minimize failing cases, the four-rules-of-simple-design analog (commutativity, associativity, idempotence, round-trip, oracle, invariant), the difference between example-based tests (one input, one assertion) and property-based tests (many generated inputs, one universal claim), why property tests find bugs example tests don't, the shrinking discipline that produces minimal failing cases, and the trade-off between generator complexity and bug-finding capacity. Do NOT use for specifying one concrete behavior with one input (use example-based tests under testing-strategy), for fuzz-testing focused on crashes (use fuzz-testing), for mutation testing as a test-suite quality signal (use mutation-testing), or for model-based testing of state machines (use state-machine-modeling)."
# license: SPDX-compatible license identifier for the skill content.
license: MIT
allowed-tools: Read Grep
# metadata: Skill Metadata Protocol fields encoded under Agent Skills-compatible frontmatter.
metadata:
# schema_version: protocol contract version this skill conforms to.
# Integer 8. Prior contract retrievable via `git show schema-v7:schemas/skill.schema.json`.
# version: skill content version (semver). Bumped when the instructional content changes.
# === v8 Classification (subject + public; polyhierarchy via subjects[]) — see ADR-0020 ===
# subject: primary browse shelf — what the skill teaches. One of twelve closed values:
# backend-engineering / frontend-engineering / software-architecture / data-engineering / agent-ops / ai-engineering /
# quality-assurance / design / reasoning-strategy / software-engineering-method / knowledge-organization / product-domain.
subject: quality-assurance
# public: publishability / private-data gate. true = safe for public release; false = private/internal.
public: true
# scope: required free-text statement of what this skill teaches and what it does not.
scope: "Use when reasoning about tests that specify universal properties of code rather than specific input-output pairs: the forall(input) → property quantification, the generator/shrinker primitives that produce inputs and minimize failing cases, the four-rules-of-simple-design analog (commutativity, associativity, idempotence, round-trip, oracle, invariant), the difference between example-based tests (one input, one assertion) and property-based tests (many generated inputs, one universal claim), why property tests find bugs example tests don't, the shrinking discipline that produces minimal failing cases, and the trade-off between generator complexity and bug-finding capacity. Do NOT use for specifying one concrete behavior with one input (use example-based tests under testing-strategy), for fuzz-testing focused on crashes (use fuzz-testing), for mutation testing as a test-suite quality signal (use mutation-testing), or for model-based testing of state machines (use state-machine-modeling)."
# public: publishability / private-data gate. true = safe for public release; false = private/internal.
# Project anchoring lives in project[] and requires grounding when present.
# taxonomy_domain: optional hierarchical sub-path within `subject`. Slash-delimited
# lowercase kebab-case segments. rename of the original v8 `domain`. Remove when the flat
# `subject` is sufficient.
taxonomy_domain: quality/testing
# owner: team handle, GitHub username, or tool name responsible for keeping this skill current.
# freshness: ISO date the skill body was last reviewed or updated.
# drift_check: truth-source verification record. Object with required `last_verified`
# (ISO date) and optional `truth_source_hashes`. Record hashes with:
# `node scripts/skill-graph-drift.js --record --apply <skill-dir>`.
# === Evaluation Status: three orthogonal axes ===
# eval_artifacts: disk-truth — does an eval file exist on disk?
# none (no intent) / planned (intent declared, no file yet) / present (file exists).
# eval_state: runtime-truth — has the eval been run and passed?
# unverified (no run yet, or no file) / passing (one-shot green) / monitored (cadenced green).
# `monitored` is strictly stronger than `passing` — a forward state for continuous runs.
# routing_eval: routing-coverage — is the skill's activation verified by the harness?
# absent (not verified) / present (gated by lint check 12; harness must exit 0).
# comprehension_state: marker that this skill has populated v6+ Understanding fields
# (mental_model, purpose, boundary, analogy, misconception). Value: `present` or absent.
# stability: lifecycle marker. One of:
# experimental (active development) / stable (production-ready) /
# frozen (no further changes expected) / deprecated.
# When `deprecated`, schema's allOf REQUIRES `superseded_by: <real-skill-name>`.
stability: experimental
# keywords: semantic phrases for fuzzy router activation. v8 cap: max 10.
# Keep terms a user would actually type when starting a task in this skill's domain.
keywords: ["property-based testing","PBT","QuickCheck","Hypothesis","fast-check","generator","shrinker","forall","invariant","round-trip property"]
# triggers: explicit-match activation phrases the router fires on literally.
# Use when label-based routing is intended; usually keywords + examples are enough.
triggers: ["should this be a property test","what's an invariant for this function","the bug only happens on weird inputs","QuickCheck or fast-check","how do we test all possible cases"]
# examples: 2-5 realistic user prompts the skill SHOULD activate for.
# Written in the user's voice. Improves retrieval recall beyond keywords alone.
examples: ["design property-based tests for a sorting function that exercise the universal contract","decide which functions in a parser deserve property tests vs example tests","diagnose a property test that finds a real bug but the shrunk input is large — likely a poorly-shrinkable generator","explain the round-trip property for an encode/decode pair"]
# anti_examples: near-miss prompts that should route ELSEWHERE.
# Pair with relations.suppresses to indicate the confusable territory's owner.
anti_examples: ["specify one concrete input-output case (use example-based tests; see testing-strategy)","fuzz for crashes and memory safety (use fuzz-testing)","measure whether tests would catch a defect (use mutation-testing)"]
# relations: typed graph edges to sibling skills. Six edge types:
# related (adjacency for browse / co-routing expansion) /
# suppresses (exclude listed skills from co-routing when THIS skill wins;
# write reason as "I own this exclusively over X", not "use X instead") /
# verify_with (cross-check; co-loaded as one-hop expansion) /
# depends_on (composition; transitive — A→B→C loads all three) /
# broader / narrower (SKOS-style generalization; broader drives co-load, narrower does not).
# === Understanding fields (when comprehension_state: present) ===
# mental_model: the primitives of the concept and how they relate. One paragraph.
mental_model: |
Property-based testing changes the unit of specification from the example to the universal contract. The four primitives are: the *property* (a forall-quantified claim that must hold for all inputs in a domain), the *generator* (produces random inputs spanning the domain including its edges — empty, single-element, boundary values, Unicode pathologies), the *shrinker* (when the property fails, reduces the failing input to its minimal demonstration — without shrinking, a failing input is random noise; with shrinking, it is the smallest case that exhibits the bug), and the *trial budget* (how many inputs the framework tries before declaring the property held — 100 for cheap properties, 1000+ for slow code or rare-bug input spaces, nightly long-running campaigns for production-critical properties).
Three classical property shapes: *invariant* (`forall L. isSorted(sort(L))` — output has a structural property regardless of input), *oracle* (`forall L. fastSort(L) == referenceSort(L)` — output equals an alternative implementation's), and *round-trip* (`forall x. decode(encode(x)) == x` — a transformation composed with its inverse is identity). Plus *algebraic* patterns (commutativity, associativity, identity, idempotence) and *metamorphic* patterns (relations between related inputs). The QuickCheck heritage (Claessen & Hughes 2000) and modern tools — Hypothesis (Python), fast-check (JS/TS), ScalaCheck, proptest, jqwik (JVM) — make the technique practical across ecosystems with strong default generators and shrinkers for primitives and collections.
# purpose: the problem this concept solves and why the field exists. One paragraph.
purpose: |
Replaces hand-written example sets with generative tests that exercise the *universal contract* of the code. Solves the limitation that example tests verify only the specific cases the author thought to write down — they miss the edge cases the author didn't think of, the combinations they couldn't enumerate, and the boundary conditions hidden in the input space. A well-designed property test exercises hundreds or thousands of inputs per run, including the edges humans systematically miss (empty collections, single-element collections, off-by-one boundaries, Unicode edge cases, integer overflow zones). When the bug is rare in the hand-written example space but findable in the generator's reach, property tests catch what examples cannot. Industrial validation: Hughes et al. (2016) caught real Dropbox synchronization bugs via PBT that example tests had not surfaced over years.
# boundary: what this concept is NOT. Distinguishes from adjacent skills by naming the
# MECHANISM that differs, not just the label. Universal terms only — no repo-specific nouns.
concept_boundary: |
Distinct from testing-strategy, which owns the strategic question of what test levels to invest in — this skill is one tactical technique within the strategy, complementary to example-based tests, not a replacement. Distinct from mutation-testing, which measures whether the test suite catches code-level defects — the two compose: PBT tends to produce high-mutation-killing tests because universal properties are specific about behavior across many inputs; mutation testing then measures their effectiveness. Distinct from type-safety, which constrains the input space at compile time — stronger types reduce the property-test surface needed; PBT is most valuable where types alone cannot encode the invariants (algorithmic correctness, business rules, round-trips). Distinct from fuzz-testing, which asserts no-crash implicitly — PBT asserts an explicit property. Distinct from state-machine-modeling, which owns finite-state representation of workflows (stateful PBT is a related narrower technique). Distinct from test-driven-development, which can be done with either example or property tests (the schools and the unit-of-specification choice are orthogonal).
# analogy: one-sentence metaphor preserving the core mechanism.
analogy: "A property-based test is to an example test what an actuarial table is to a single insurance claim — the example tells you what happened in one case, the property tells you what must hold across the entire population; you do not learn that fire insurance is sound by inspecting one policy, you learn it from a contract that quantifies over every house ever insured."
# misconception: the wrong mental model people bring; corrected explicitly.
misconception: |
The wrong mental model is that property tests *replace* example tests, or that any test with random input is a property test. They do not, and it is not. A test that runs `forall input in [single_value]` is an example test in property-test clothing. A "property" written after the fact to fit whatever the generator happens to produce is not a universal claim — it is whatever the generator's reach happens to be, which verifies nothing meaningful. Property tests are *complements* to example tests in a mature suite: properties for the universal contract; examples for specific cases that document particular behaviors or bug fixes. The discipline lives in *three* places simultaneously — the property must be universal, falsifiable, and meaningful (writing a property requires articulating the contract; if the contract is not articulable, PBT does not apply); the generator must produce inputs that cover the space *including the edges* (generators that produce only "typical" inputs miss bug-triggering edges by construction); the shrinker must reduce failing inputs to a minimal form (if the reported failure is a 500-element random list, the generator/shrinker pair is the bug, not the property). All three must be well-designed for PBT to deliver its promise.
# concept: legacy v5 nested Understanding block. DEPRECATED — flat fields above
# (mental_model, purpose, boundary, analogy, misconception) win when both are present.
# === Export provenance (set by the export pipeline; do not hand-author) ===
# skill_graph_protocol is a content-label claim distinct from `schema_version` semantics.
# See AGENTS.md § Version Labels Are Earned, Not Bumped.
skill_graph_source_repo: "https://github.com/jacob-balslev/skill-graph"
skill_graph_project: Skill Graph
skill_graph_canonical_skill: skills/quality-assurance/property-based-testing/SKILL.md
# === Health Block (written by the audit loop, not hand-authored) ===
# See SKILL_AUDIT_LOOP.md § The Health Block. UNVERIFIED is the honest default.
#
# structural_verdict: form/export shape (gates 1-2, 7 — external mandates only).
# PASS / PASS_WITH_FIXES / FAIL / UNVERIFIED.
# truth_verdict: truth sources vs declared hashes (gates 3-6).
# PASS / DRIFT / BROKEN / UNVERIFIED.
# comprehension_verdict: gate 8 — cheap recitation smoke test. Never alone certifies.
# PASS / SHALLOW / REDUNDANT / UNVERIFIED / PROVISIONAL / SKIPPED_BASELINE_HIGH / NA.
# application_verdict: gate 9 — the primary quality signal. APPLICABLE is the only verdict
# that certifies the skill is USEFUL (grader-confirmed). PROVISIONAL = one model self-assessed.
# APPLICABLE / REDUNDANT / HARMFUL / MIXED / FALSE_POSITIVE / PROVISIONAL / UNVERIFIED.
relations:
related: ["test-driven-development","testing-strategy","type-safety","mutation-testing"]
suppresses: ["testing-strategy","mutation-testing","type-safety"]
verify_with: ["testing-strategy","mutation-testing"]
Property-based testing changes the unit of specification from the example to the universal contract. The four primitives are: the property (a forall-quantified claim that must hold for all inputs in a domain), the generator (produces random inputs spanning the domain including its edges — empty, single-element, boundary values, Unicode pathologies), the shrinker (when the property fails, reduces the failing input to its minimal demonstration — without shrinking, a failing input is random noise; with shrinking, it is the smallest case that exhibits the bug), and the trial budget (how many inputs the framework tries before declaring the property held — 100 for cheap properties, 1000+ for slow code or rare-bug input spaces, nightly long-running campaigns for production-critical properties).
Distinct from testing-strategy, which owns the strategic question of what test levels to invest in — this skill is one tactical technique within the strategy, complementary to example-based tests, not a replacement. Distinct from mutation-testing, which measures whether the test suite catches code-level defects — the two compose: PBT tends to produce high-mutation-killing tests because universal properties are specific about behavior across many inputs; mutation testing then measures their effectiveness. Distinct from type-safety, which constrains the input space at compile time — stronger types reduce the property-test surface needed; PBT is most valuable where types alone cannot encode the invariants (algorithmic correctness, business rules, round-trips). Distinct from fuzz-testing, which asserts no-crash implicitly — PBT asserts an explicit property. Distinct from state-machine-modeling, which owns finite-state representation of workflows (stateful PBT is a related narrower technique). Distinct from test-driven-development, which can be done with either example or property tests (the schools and the unit-of-specification choice are orthogonal). A property-based test is to an example test what an actuarial table is to a single insurance claim — the example tells you what happened in one case, the property tells you what must hold across the entire population; you do not learn that fire insurance is sound by inspecting one policy, you learn it from a contract that quantifies over every house ever insured. The wrong mental model is that property tests replace example tests, or that any test with random input is a property test. They do not, and it is not. A test that runs forall input in [single_value] is an example test in property-test clothing. A "property" written after the fact to fit whatever the generator happens to produce is not a universal claim — it is whatever the generator's reach happens to be, which verifies nothing meaningful. Property tests are complements to example tests in a mature suite: properties for the universal contract; examples for specific cases that document particular behaviors or bug fixes. The discipline lives in three places simultaneously — the property must be universal, falsifiable, and meaningful (writing a property requires articulating the contract; if the contract is not articulable, PBT does not apply); the generator must produce inputs that cover the space including the edges (generators that produce only "typical" inputs miss bug-triggering edges by construction); the shrinker must reduce failing inputs to a minimal form (if the reported failure is a 500-element random list, the generator/shrinker pair is the bug, not the property). All three must be well-designed for PBT to deliver its promise.
The tactical testing technique in which a test specifies a universal property — a claim that must hold for all inputs in a domain — and a framework generates many random inputs to challenge the claim, shrinking any failing input to its minimal form. Covers the four primitives (property, generator, shrinker, trial budget), the three classical property shapes (invariant, oracle, round-trip) plus algebraic and metamorphic patterns, the QuickCheck heritage and the modern tool ecosystem (Hypothesis, fast-check, ScalaCheck, proptest, jqwik), the generator/shrinker discipline for domain types, and the integration of property tests with example tests in a test suite.
Property-based testing changes the unit of specification from the example to the contract. An example test says "for this input, the output should be this." A property test says "for any input matching this generator, this universal claim holds." When the contract is articulable, a property test verifies far more of the behavior than a hand-written set of examples could; when the contract is not articulable, property tests are made up after the fact and verify whatever happens to be in the generator's reach, which is not useful.
The discipline is in three places: the property (is the claim universal, falsifiable, and meaningful?), the generator (does it produce inputs that cover the space, including the edges?), and the shrinker (when a property fails, can the framework reduce the failing input to something a developer can read?). All three must be well-designed for property-based testing to deliver its promise.
Property tests are complements to example tests, not replacements. The mature pattern is properties for the universal contract and examples for the specific cases that document particular behaviors or bug fixes.
Plus algebraic patterns (commutativity, associativity, identity, idempotence) and metamorphic patterns (relations between related inputs). Most useful property tests use one or two of these shapes.
A property test has two halves: the property and the generator. A well-designed generator produces inputs that exercise the input space — including edge cases (empty, single-element, boundary values), valid cases, and invalid cases as appropriate. A poorly-designed generator misses bug-triggering inputs and the property "passes" by not having been challenged.
The shrinker reduces a failing input to its minimal form. Without shrinking, a failing input is random noise. With shrinking, a failing input is the smallest demonstration of the bug — usually small enough that the bug is visible at a glance.
A test suite typically mixes both; the right ratio depends on how much of the code has articulable universal contracts.