---
title: Self-healing documentation workflows
description: Detecting and fixing documentation drift against a source of truth
  using multi-agent verification loops.
date: 2026-10-30
tags:
  - ai
  - agents
  - documentation
url: https://skurekjakub.dev/blog/self-healing-docs
author: Jakub Skurek
---

> Agent instructions:
> Take page paths from the index rather than constructing them — guessed paths 404. Links inside this document already point at `.md`; following them, or sending `Accept: text/markdown`, keeps you in Markdown.
> When citing this page, use the `url` in its frontmatter — the HTML page; this `.md` address is a rendering of it.
>
> - [llms.txt](https://skurekjakub.dev/llms.txt) — every page with a Markdown twin, with descriptions.
> - [skill.md](https://skurekjakub.dev/skill.md) — how to read this site as an agent.

Documentation tends to drift when the product changes (a default value is updated, a setting moves to a different menu, or a configuration key is removed). When [Tan, Wagner, and Treude](https://arxiv.org/abs/2212.01479) scanned over 3,000 GitHub projects for references to deleted or renamed code elements, most projects had carried broken references at some point in their history.

Standard documentation tools do not catch factual drift:

- **Prose linters** ([Vale](https://vale.sh/)) check style, grammar, and terminology, not whether the software still works as described.
- **Procedure runners** ([Doc Detective](https://docs.doc-detective.com/)) run documented steps against a live environment, but require writing and maintaining test specs first.
- **Unconstrained agent sweeps** (asking a single LLM to "update outdated pages") tend to rewrite phrasing, reorganise headings, and approve their own hallucinations.

## Design principles

When documentation sits next to a machine-readable source of truth (product source code, an OpenAPI spec, a GraphQL schema, or CLI `--help` output), factual claims can be checked directly. If a page states that logs are written to `App_Data\logEvents.log` and the source code at the current release tag writes them elsewhere, the page is wrong.

To turn those checks into reviewable pull requests, a verification loop needs a few constraints:

- **Target provable facts only.** The workflow finds discrepancies, verifies them against the source of truth, and opens a pull request. It never pushes to `main` and ignores style, wording, page structure, and clarity. Anything the source code cannot settle is out of scope.
- **Verify against a shipped release tag.** Documentation describes what users can install today, not the development branch. Claims are checked against a pinned release tag. If a page matches the release tag but differs from `master`, it is tagged `upcoming` for future release notes rather than fixed prematurely.
- **Tag error depth to track convergence.** Grouping findings by depth (`typo`, `name`, `value`, `behaviour`, `omission`) provides a stopping signal. Initial runs catch shallow renames and changed defaults; later runs shift toward behavioural claims until searches consistently return nothing.
- **Blind the validator.** An LLM given both a claim and the code snippet that supposedly proves it will usually agree. Language models [favour stated premises](https://arxiv.org/abs/2310.13548), [struggle to self-correct without external grounding](https://arxiv.org/abs/2310.01798), and [prefer outputs that look like their own](https://arxiv.org/abs/2404.13076). The agent verifying a claim should only receive the quoted sentence and its file location, forcing it to check the codebase from scratch.
- **Open one pull request per fact.** Fixing a single factual claim per PR gives the reviewer one yes-or-no question to verify, backed by source citations, git history, and screenshots.

## Implementation: Xperience by Kentico

To see how these rules work in practice, here is a proof-of-concept pipeline built for the [Xperience by Kentico](https://docs.kentico.com) documentation repository. The docs repo vendors the C# product source as a local git clone, so agents can inspect release tags and commit history on disk.

> **Note:** This pipeline runs in an internal repository, so the pull requests are not public. The prompts, schemas, and numbers below are taken from the production workflow.

### Pipeline and subagents

The coordinator is a [Claude Code skill](https://code.claude.com/docs/en/skills) using the orchestrator-workers pattern from Anthropic's [Building effective agents](https://www.anthropic.com/engineering/building-effective-agents). The orchestrator does not read product code or documentation diffs itself; it spawns isolated subagents sequentially, reads their handoff files from disk, and logs outcomes in `ledger.md`.

![Flowchart of a single finding through the pipeline: the Finder reads ledger.md and checks claims against the release tag and master. Undocumented or upcoming items go to deferred.md. Wrong or missing claims are verified on origin/main, passed to a blind Validator, checked in the running app by Runtime, edited on a fresh branch by the Fixer, checked by the Fix reviewer, and opened as a single PR per fact.](https://skurekjakub.dev/blog/self-healing-docs/heal-loop.drawio.svg)

_One finding from start to finish. Every exit path writes a row back to ledger.md._

Each finding passes through specific subagent roles:

- **Finder:** Reads `ledger.md`, picks an in-scope page not yet marked clean, and checks its claims against the release tag and `master`:
  ```bash
  # Inspect product source at the release tag, not the working tree
  git -C resources/repositories/xperience grep -n CMSPhysicalFilesCacheMinutes 31.10.0
  git -C resources/repositories/xperience show 31.10.0:CMSSolution/EventLog/EventLogHelper.cs
  ```
- **Validator:** Receives only the documentation file path, line number, and quoted claim. It checks the claim from a blank context against the release tag and collects the git and source evidence if confirmed.
- **Runtime:** Starts a local sample site at the release tag to reproduce the finding in the running app (screenshots, HTTP traces, or files on disk). Skipped for code-only claims.
- **Fixer:** Checks out a fresh branch from `origin/main` and updates all occurrences of the wrong fact across the docs.
- **Fix reviewer:** Checks the commits on the fix branch against the validator's verdict, approving the branch for a PR or returning it with notes.

### Handoff schema and ledger state

Subagents write results to `.cache/docs-heal/handoffs/` in a fixed format so the orchestrator only has to read the header fields to route the next step:

```markdown
# H-001 finder

- verdict: finding | none
- kind: wrong | missing | undocumented | upcoming
- depth: typo | name | value | behaviour | omission
- page: <path>:<line>
- claim: "<quoted documentation text>"
- correct: <what the source at the tag does>
- evidence at tag: <path>:<line> — "<quoted code>"
- evidence at master: same | <path>:<line> — "<quoted code>"
```

`ledger.md` tracks every claim raised so far and every page that checked out clean. The finder receives the same prompt on every iteration and relies on the ledger to avoid repeating work:

```text
Finding H-006. Ledger: .cache/docs-heal/ledger.md. Release tag: 31.10.0. Handoff: .cache/docs-heal/handoffs/H-006-finder.md.
```

### Validator blinding and evidence

Because each [Claude Code subagent](https://code.claude.com/docs/en/sub-agents) runs in a separate context window, the orchestrator can filter what the validator sees. It withholds the finder's `correct` summary and code citations, verifies via `git grep` on `origin/main` that the quoted sentence exists, and passes only the bare claim and location to the validator:

![Diagram showing information filtering between Finder and Validator: the Finder handoff contains page:line, claim, candidate, plus withheld fields (correct, evidence at tag, evidence at master). Only page:line, claim, and candidate pass through the Orchestrator into the Validator prompt, which runs in a fresh context.](https://skurekjakub.dev/blog/self-healing-docs/validator-blind.drawio.svg)

_Validator prompt and context._

If the finder misread a fallback or missed an override, the validator rejects the finding (`page-correct`, `unreleased`, `overridden`, `not-provable`, or `out-of-scope`).

When the validator confirms an error, it collects the evidence block for the pull request:

- **Source at the tag:** File paths and line numbers at the release tag, plus whether `master` matches.
- **Tests and doc comments:** XML doc comments or unit tests in the product repo asserting the behaviour.
- **Product git history:** The commit that introduced the behaviour (`git log -S`) and the first release tag containing it (`git tag --contains`).
- **Tickets and release notes:** The Jira issue from the product commit and whether release notes covered the change.
- **Docs git history:** The commit that introduced the wrong claim in the docs.
- **Cross-references:** Other documentation pages that state the fact accurately or need the same update.

The PR description stays brief, and the full evidence block is posted in a resolved comment thread.

### Runtime proofs and deferred items

When a confirmed finding is visible in the application (an admin UI label, default value, file on disk, or HTTP header), the runtime subagent checks out the sample project at the release tag, drives the UI through a headless browser CLI, and captures screenshots for the PR.

Findings tagged `undocumented` or `upcoming` skip the validator and PR flow and are written to `deferred.md` as `pending`. Only a maintainer updates deferred statuses (`won't do: <reason>`, a Jira key, or `fixed in <PR>`).

### Operating modes

The workflow runs in either broad mode or commit mode:

- **Broad mode:** The finder checks any in-scope page not marked clean in `ledger.md`. A run stops after opening five PRs or after five consecutive finders return `none`.
- **Commit mode:** The orchestrator diffs `main` from a stored commit hash (`git diff --name-only <marker>..origin/main`) and limits the finder to pages modified in that range.

![Comparison of Broad Mode and Commit Mode: Broad mode feeds every in-scope page to the Finder and stops at 5 PRs or 5 empty finders. Commit mode feeds git diff --name-only marker..origin/main to the Finder and advances the marker in ledger.md when the range is done. Both share ledger.md.](https://skurekjakub.dev/blog/self-healing-docs/heal-modes.drawio.svg)

_Broad mode and commit mode._

### First-run telemetry

The first broad-mode run against release tag `31.10.0` ran five finders, confirmed all five discrepancies, and opened five pull requests with zero rejections:

| ID      | Depth   | Discrepancy                                         | Runtime check                  | Total tokens |
| :------ | :------ | :-------------------------------------------------- | :----------------------------- | -----------: |
| `H-001` | `value` | Default file path for the event log                 | —                              |      189,662 |
| `H-002` | `name`  | Admin settings menu path for URLs and SEO           | —                              |      187,815 |
| `H-003` | `name`  | Configuration key removed from the product          | —                              |     ≥266,821 |
| `H-004` | `value` | Image file extensions allowed for upload by default | Shown via Content hub uploader |     ≥354,673 |
| `H-005` | `name`  | ASP.NET Core Tag Helper attribute name              | Skipped (code-only)            |      157,113 |

Total usage across the run was at least 1,156,084 tokens over 543 tool calls and 57 minutes of subagent execution, averaging \~230K tokens per pull request. Anthropic reported a similar token overhead in their [multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system) due to isolated context windows.

Token usage by subagent role:

- **Finders:** 38K–82K tokens
- **Validators:** 45K–99K tokens
- **Runtime:** 147K tokens
- **Fixers and reviewers:** 19K–47K tokens

A five-for-five hit rate on shallow `name` and `value` errors is typical on an initial pass over an existing documentation set. In later runs, once obvious renames are cleared out, findings shift toward `behaviour` and `omission` and the validator begins rejecting borderline claims.

## Conclusion

Most of the reliability in this workflow comes from the harness. Models already spot suspicious claims well; keeping the output mergeable comes down to checking a pinned release tag, restricting scope to binary facts, running validators in a blank context window, and opening one pull request per fact with the source evidence attached.

---

Sources:

- [Detecting Outdated Code Element References in Software Repository Documentation](https://arxiv.org/abs/2212.01479) — Tan, Wagner, and Treude (2022)
- [Towards Understanding Sycophancy in Language Models](https://arxiv.org/abs/2310.13548) — Sharma et al. (2023)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://arxiv.org/abs/2310.01798) — Huang et al. (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://arxiv.org/abs/2404.13076) — Panickssery, Bowman, and Feng (2024)
- [How we built our multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system) — Anthropic Engineering
- [Building effective agents](https://www.anthropic.com/engineering/building-effective-agents) — Anthropic Engineering
- [Claude Code subagents](https://code.claude.com/docs/en/sub-agents) and [skills](https://code.claude.com/docs/en/skills) documentation
- [Vale prose linter](https://vale.sh/) and [Doc Detective](https://docs.doc-detective.com/)
