Lorna Mitchell’s article on treating documentation like code deserves credit for translating a powerful engineering workflow into an approachable practice: use plain text, version control, pull requests, review, automation, and repeatable publication instead of leaving documentation as an isolated final task.
That workflow solves important provenance and collaboration problems. It does not, by itself, establish that the documentation is correct. A page can compile, pass a style checker, and deploy successfully while giving the reader an obsolete command or an impossible sequence of steps.
Documentation as code becomes reliable only when it also gains tests appropriate to documentation.
The toolchain verifies the artifact, not necessarily the claim
A typical documentation pipeline can check frontmatter, syntax, spelling, internal links, formatting, and whether a static-site generator completes. These checks are valuable because they convert recurring editorial mistakes into immediate feedback.
But most documentation makes behavioural claims:
- this command installs the tool;
- this option has a particular effect;
- these API fields are required;
- this sequence creates a working deployment;
- this example produces the displayed result;
- this version remains compatible with another component.
A Markdown parser cannot validate those claims. Treating build success as documentary correctness repeats the same mistake as treating successful compilation as proof that software meets its requirements.
The source-controlled text and its described system must remain connected by executable or inspectable evidence.
A layered test model for documentation
Different failures require different checks:
| Layer | Example check | What it can establish |
|---|---|---|
| Structure | frontmatter and navigation validation | the page can be indexed and rendered |
| Language | spelling, terminology, and style rules | wording follows agreed conventions |
| References | internal and external link checks | referenced locations are reachable |
| Code | compile or run extracted snippets | examples are syntactically and operationally valid |
| Interface | compare documented options with schemas or CLI help | named parameters still exist |
| Procedure | execute instructions in a clean environment | the documented path reaches the claimed state |
| Reader task | usability review with the intended audience | a person can understand and complete the task |
| Freshness | ownership and review-expiry checks | silent staleness becomes visible |
No pipeline needs every layer for every page. A conceptual essay does not require a containerized installation test. A security-sensitive recovery procedure does. The strength of the check should follow the consequence of documentary failure and the cost of obtaining evidence.
Executable examples should be single-sourced where possible
Copied code drifts. A command in a README, tutorial, example project, and test fixture can become four subtly different interfaces. When practical, documentation should include or extract examples from artifacts that are compiled and tested with the product.
Useful patterns include:
- running fenced code blocks as part of CI;
- generating reference pages from authoritative schemas;
- importing tested example files into documentation;
- building tutorials against disposable containers;
- checking API requests against a versioned contract;
- recording the product and dependency versions used for each procedure.
Generation is not always the answer. Automatically produced reference material can be complete yet unreadable, while hand-written explanations can express intent and trade-offs more clearly. The goal is not to eliminate prose. It is to avoid two independent sources for facts that machines can keep aligned.
Procedure tests need clean state and explicit outcomes
Installation and operational guides often appear to work because the author’s machine contains unmentioned credentials, cached dependencies, environment variables, or configuration from an earlier attempt. A credible procedure test starts from a declared state and checks a declared result.
For example:
- create a clean environment from a pinned image;
- execute only the documented prerequisites and commands;
- capture output and exit status;
- verify the promised service, file, API response, or observable state;
- repeat the teardown and recovery path when those are documented;
- preserve logs so a failure can be distinguished from infrastructure noise.
This converts a tutorial from a plausible narrative into a reproducible path. It also reveals missing assumptions—the most common class of documentation defect.
Human review remains a different oracle
Automation can show that a command runs. It cannot fully determine whether the reader knows when to use it, understands its risk, or can recover when reality differs from the happy path. Documentation review should therefore separate at least three questions:
- Technical accuracy: does the described system behave this way?
- Task completeness: can the intended reader reach the required outcome?
- Explanatory quality: are purpose, constraints, alternatives, and failure modes understandable?
Subject-matter experts are good at the first question but often fill gaps from experience. New or representative readers are better at exposing missing context. Editors can detect ambiguity that neither the code nor the original author notices. These roles produce different evidence and should not be collapsed into one approval.
Staleness is an ownership failure
Documentation commonly fails after publication rather than before it. The responsible component changes, but no signal reaches the page owner. Docs-as-code helps because code and documentation can share a repository and review process, yet proximity does not guarantee coordinated change.
Useful controls include:
- ownership metadata for high-risk pages;
- links from documentation to the code or schema it describes;
- change checks that identify affected docs;
- explicit version applicability;
- review dates based on volatility and consequence;
- analytics and feedback channels that reveal failed reader tasks;
- deletion or archival when a page can no longer be maintained.
An old review date is not proof of an error, but it is a visible uncertainty. Silent permanence is worse.
Mitchell’s core recommendation remains sound: documentation benefits from the same collaborative discipline as software. The stronger conclusion is that the analogy must continue beyond Git and pull requests. Software is tested against behaviour; consequential documentation should be tested against the tasks and interfaces it claims to describe. A green documentation build proves that a document was produced. Reliability begins when the pipeline also asks whether the document is still true.