Verify AI-Generated Docs Against the Code Before You Merge Them

An assistant can draft a README, an API page, or a migration note that reads like your project and still describes a flag you do not have, a default you changed, or an error you never throw.

The draft is useful. The merge is the dangerous step. Treat generated documentation like a pull request from a fast new teammate who has not run the code: you verify it against the repo, then you merge it.

This is the developer companion in a short cluster on documentation drift. A Weekly Drift Check for Product Docs covers a weekly check of product docs against the live product. A Telegraph note covers writing a one-page brief before you ask for a README draft.

Generate from the right inputs

Give the assistant the files that actually define the behaviour: the module, the OpenAPI spec, the flags, the tests. A marketing page or an old README will teach it the stale version.

Ask for a draft of a specific section, not “document the whole service.” The smaller the scope, the cheaper the check.

A verification pass that fits in a review

Work through the draft with the repo open.

• Names. Every command, flag, environment variable, route and field must appear in the code or the spec, with the same spelling.
• Defaults and limits. If the draft says a timeout is 30 seconds, find the constant. If it says a plan allows 10,000 rows, find the check.
• Examples. Run them, or at least compile them. Invented snippets are common and very costly.
• Errors and edge cases. If the page lists error codes, grep for them. Missing ones are a problem; extra ones are also a problem.
• Scope. Delete advice that belongs in another repo, or that describes a roadmap as if it had shipped.

If you cannot point to a file for a sentence, cut the sentence or mark it as unknown. Do not leave a confident guess in the main branch.

What tools can and cannot do

Some documentation tools are built to notice when a page and a repo have drifted, or to open a draft from a diff. That can shrink the search. It does not replace the pass above. A useful starting point, if you want a shortlist, is Toolbit’s comparison of AI documentation sync tools, which is organised around a drift test. Run any candidate on a page you already know is wrong, and see whether it flags the same mismatch you found.

The review still needs a named author, the same way a code PR does. If nobody can explain the page without re-reading the model’s prose, it is not ready.

Checklist

• The draft was generated from current code or a current spec, not only from an old README.
• Every command, example and default has been checked against the repo.
• Invented or out-of-scope lines have been removed.
• A person who owns the code is listed on the change.

Takeaway

Merge AI-generated docs only after you have checked them against the code. Fluency is not evidence that a flag, an example, or an error code is real.