My original LinkedIn post recommended Mourjo Sen’s InfoQ article, Beyond Accidental Quality. Sen deserves explicit credit not only for covering generators, shrinking, invariants, and unknown unknowns, but for connecting them in an explanation that remains accessible without flattening the engineering problem. That combination of depth and clarity is why the article impressed me and why I shared it.
This commentary starts from Sen’s account and develops one architectural consequence. “Generate more inputs” is not the central idea of property-based testing.
The central idea is to state an executable claim about a system and then search for a counterexample. A generator controls where that search goes. An oracle determines what counts as failure. A shrinker turns a discovered failure into an inspectable case. None of these components is sufficient alone.
Property-based testing therefore starts with the property, not the generator.
What property-based testing promises
The original QuickCheck paper described properties as executable functions that can be checked automatically against randomly generated input. It also allowed custom generators for domains where the default distribution was unsuitable.
This model changes the unit of test design:
- an example test claims that one selected input has one expected result;
- a property claims that a relationship should hold across a defined input domain;
- the test engine attempts to falsify that relationship by producing inputs.
The distinction is important, but it must not be exaggerated. Passing a finite number of generated cases is not a proof that the property holds universally. It is evidence bounded by the property, the generator, its distribution, the number of attempts, and the observability of the test harness.
The property is the real test design
A sophisticated generator attached to a weak property creates activity, not assurance. Consider a sort function. The assertion that its output is ordered is necessary but incomplete: a function that always returns an empty list satisfies it. Useful testing also needs claims about element preservation, multiplicity, and perhaps idempotence.
Several property patterns repeatedly provide leverage:
| Pattern | Example claim | Principal risk |
|---|---|---|
| Invariant | A transfer preserves the total balance | Important state may be omitted from the invariant |
| Round trip | Decoding an encoded value recovers the original | Encoder and decoder may share the same mistake |
| Differential | Two independent implementations return equivalent results | The reference may be wrong or not truly independent |
| Metamorphic | A controlled input transformation preserves a relationship | The relation may cover only one behaviour class |
| Model or state machine | Every command sequence conforms to an abstract model | The model may reproduce the implementation defect |
These are not templates that automatically become correct tests. They are ways to expose the oracle design. A good property says which behaviour matters and why a generated counterexample would refute the claim. A bad one mirrors the implementation, checks a tautology, or ignores the state in which the expensive failures occur.
Random does not mean broad
Randomized generation can escape the handful of examples a person happened to imagine. It does not remove human choice; it moves that choice into the generator and its distribution.
A generator may produce valid values while almost never reaching the boundary combinations that matter. Filtering arbitrary values through a demanding precondition can be even worse: most attempts are discarded, so the apparent test count says little about exercised behaviour. Coverage-guided property-based testing was motivated in part by this sparse-precondition problem.
The relevant question is not “How many random cases ran?” It is “Which meaningful regions of the input and state space were sampled, and with what frequency?” A defensible generator design should make visible:
- validity constraints and discarded-input rates;
- equivalence classes and important boundaries;
- size and structural-complexity distributions;
- correlations between fields rather than independent noise;
- rare states that require constructive generation;
- coverage or domain classifications that reveal sampling bias.
This is where PBT meets test architecture. The generator is an executable model of the test domain. Its mistakes can systematically hide faults even while thousands of cases pass.
Shrinking is part of the evidence
Finding a failure is only the first step. A large generated object or long command sequence may contain enough irrelevant structure to conceal the mechanism of failure. Shrinking searches for a smaller case that still falsifies the property.
The value is not cosmetic. A minimal or near-minimal counterexample can separate the causal structure from accidental input noise, improve defect diagnosis, and become a stable regression example. The Hypothesis reducer, for example, reduces the sequence of choices used during generation so that simplified cases remain generatable under the encoded constraints.
Shrinking also has failure modes. A custom shrinker can leave the valid domain, become too expensive, or converge on a different manifestation of a broad property failure. The reduced case should therefore be replayed, checked against domain constraints, and examined to confirm that it preserves the behaviour being diagnosed—not merely the same boolean result.
A test architecture for PBT
Treating PBT as “add a generator to a unit test” hides the decisions that determine its strength. A more inspectable architecture separates six layers:
| Layer | Design question | Evidence to preserve |
|---|---|---|
| Domain | Which inputs and states are valid? | constraints, constructors, rejected cases |
| Property and oracle | Which claim would a counterexample refute? | executable specification and rationale |
| Generation | Where and how often will the search look? | distributions, classifications, boundary weights |
| Exploration | How are new or difficult regions reached? | seeds, coverage, state transitions, search budget |
| Reduction | How is a failing case simplified without losing meaning? | shrink trace and minimized counterexample |
| Reproduction | Can the exact failure be rerun and retained? | seed, framework version, environment, regression |
This separation makes review possible. A team can challenge the property without rewriting the generator, inspect the distribution without changing the oracle, and compare reduction strategies without confusing them with bug-finding effectiveness.
Where PBT has high leverage
The empirical study Property-Based Testing in Practice found that experienced developers often use PBT opportunistically where properties are readily available and valuable. Round trips, differential checks, transformations with conserved quantities, parsers, serialization, numerical routines, and state machines are natural candidates.
PBT is less compelling when the oracle is vague, valid data is prohibitively difficult to construct, effects cannot be isolated, or each execution is too expensive to support meaningful exploration. It also should not displace carefully chosen examples. Examples document named business cases and regressions; properties search a broader domain for violations of general claims. Mature suites use both.
The corrected conclusion
Generative testing does counter one source of accidental quality: dependence on a short, handpicked list of examples. But it does not eliminate accidental quality. A weak property, biased distribution, ineffective shrinker, or self-confirming oracle can produce a large green test run with little evidential value.
The strongest contribution of property-based testing is that it makes more of the testing argument explicit. The property exposes the claim. The generator exposes the sampled domain. The shrinker exposes a counterexample. Classification and coverage expose where the search actually went.
That is more than random test generation. It is a falsification system whose quality can itself be inspected, challenged, and improved.