# What Actually Changed in How We Measure Documentation?

The core shift is methodological. According to the arXiv paper "Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer," the authors built a roundtrip benchmark that scores a code description by whether code regenerated from that description passes the original test suite. That is a different animal from BLEU scores, human ratings, or line-count heuristics. The paper reports two findings that matter operationally: completeness, not length, drives a description's fidelity; and a description-writing prompt discovered through optimization reaches full fidelity while generalizing to unseen files only partially. The first finding kills the "write more comments" reflex. The second kills the "find the one prompt" reflex. My read: this is the first credible attempt to make documentation a testable contract. If a description can be regenerated into passing code, it is complete enough for an agent. If it cannot, no amount of prose padding saves it.

# Who Is Affected First — and Who Is Not?

The immediate beneficiaries are teams running coding agents against large, under-documented codebases — legacy monoliths, acquired repos, and internal platforms with thin docstrings. Those teams now have a falsifiable target: raise roundtrip pass rate per module. The immediate losers are vendors selling generic "context" or "codebase understanding" layers whose pitch is volume of indexed tokens. The paper's completeness-over-length finding directly undercuts token-count marketing. If the signal is completeness, a 40-token description that captures every branch condition beats a 400-token description that misses one. Developers writing docstrings for human readers are not the target audience here. The paper's benchmark is agent-facing. A description optimized for agent regeneration can look terse or redundant to a human reviewer — that is a feature, not a bug.

# What Are the Operational Tradeoffs of Adopting This?

The honest tradeoff is cost. Running a roundtrip benchmark means generating code from descriptions and executing the original tests for every module you want to measure. That is compute you are spending on evaluation, not on shipping. The second tradeoff is maintenance drag. A description that passes roundtrip today can fail tomorrow after a refactor, because the tests changed even if the code did not. Teams adopting this need a CI hook, not a one-time audit. The third tradeoff is the transfer problem. The paper's own optimizer result — full fidelity on seen files, weaker on unseen ones — means you cannot buy one prompt and apply it repo-wide. You either accept per-module tuning or you accept lower fidelity on new code.

Approach	Signal	Cost	Transfer Behavior	Best Fit
Human-written docstrings	Reviewer judgment	Low	N/A	Public APIs, onboarding
Token-count / context volume	Indexed size	Low	Broad but shallow	Vendor marketing, demos
Roundtrip benchmark (this paper)	Regenerated code passes tests	High (compute + CI)	Per-module, weak cross-file	Legacy code, agent pipelines
Optimized description prompt	Fidelity on seen files	Medium	Reported as non-transferring	Single-repo pilots
Verdict	Roundtrip benchmarking wins for teams with test coverage; the optimized prompt alone loses because it does not transfer.

# What Should Teams Actually Do Next?

Start with modules that already have strong tests. The roundtrip benchmark only works where the original test suite is trustworthy — if your tests are flaky or thin, the benchmark measures test quality, not description quality. Second, instrument the pass rate. Track roundtrip fidelity per module over time, the same way you track coverage. The paper's completeness finding suggests you should expect diminishing returns from description length and near-linear returns from covering untested branches. Third, treat the prompt as a per-repo artifact. The authors reported that their optimized description-writing prompt generalizes to unseen files only partially, so plan for local tuning rather than a universal template. Budget for that.

# What Are the Falsifiable Predictions?

1. By Q2 2027, GitHub will ship a documentation-fidelity check in Actions that regenerates code from descriptions and runs tests, mirroring the roundtrip benchmark. 2. By end of 2027, at least one coding-agent vendor (Cursor, Cognition, or Anthropic) will publish per-module roundtrip fidelity numbers as a marketing metric, forcing competitors to do the same. 3. The "optimized prompt does not transfer" result will be cited by at least three follow-up papers by mid-2027 attempting cross-file transfer via retrieval-augmented description generation.

# What Should Readers Remember?

• Completeness beats length: a short description covering every branch outperforms a long one that misses a condition.
• The roundtrip benchmark is the first test-gated definition of documentation quality for agents — adopt it where tests are trustworthy.
• The optimizer's non-transfer result means one prompt will not save a repo; per-module tuning is the realistic path.
• Token-count marketing from context vendors is now falsifiable against a published benchmark.
• CI integration, not one-time audits, is the only way to keep roundtrip fidelity from decaying after refactors.