--- title: Assess AI accessibility using afdocs description: A look at the agent-friendly docs spec - https://agentdocsspec.com/ - and how it can help us judge just how accessible a site is to agents. date: 2026-08-08 tags: - documentation - ai --- With agentic traffic soon set to surpass that of humans on the internet (or already surpassing in [some sectors](https://www.mintlify.com/blog/state-of-docs-traffic)), it's a good time to start making sure websites are prepared to not just handle agentic workflows, but also provide them with an optimized browsing experience. While the landscape changes weekly, a few things are already well established: - Don't let agents parse HTML, give them an option to request markdown instead - Serve an [llms.txt](https://llmstxt.org/) file at the app root — while not canonical yet, it seems to be gaining traction - Give an indication that markdown content is available, for example as hidden, embedded content - Structure content in a clear hierarchy driven by headings That said, being aware of the rules is one thing, being able to quickly check and verify adherence is another. I wanted something I can simply point at a site to get a quick read. Before building something myself, I decided to first check whether suitable tools already exist. And I'm glad I did. [afdocs](https://afdocs.dev/) is a tool that you can run against your site that checks for bottlenecks to agentic access and outputs a structured report. On its own, it's a simple CLI wrapper over the [Agent-Friendly Documentation Spec](https://agentdocsspec.com/) (an unofficial index of best practices and conventions agent-ready docs sites should follow), which serves as its canonical source for rules, separated into several categories such as content discoverability, markdown availability, and content structure. ## Running the tool Running the tool is as simple as calling: ```bash # This installs https://github.com/agent-ecosystem/afdocs onto your system npx afdocs check "yoursite" # http://localhost:3000, mysite.com, etc. ``` ## Reading the output Here is a sample output from one of the sites I manage: ``` Agent-Friendly Docs Check: http://localhost:3002 Timestamp: 8/8/2026, 1:36:55 PM content-discoverability ✓ llms-txt-exists: llms.txt found at http://localhost:3002/llms.txt ✓ llms-txt-valid: llms.txt follows the proposed structure (H1, blockquote, heading-delimited link sections) ✓ llms-txt-size: llms.txt is 2,646 characters (under 50,000 threshold) ✓ llms-txt-links-resolve: All 8 same-origin links resolve (8 total links) ✓ llms-txt-links-markdown: 8/8 same-origin links point to markdown content (100%) ✓ llms-txt-directive-html: llms.txt directive found in HTML of all 50 sampled pages, near the top of content ✓ llms-txt-directive-md: llms.txt directive found in markdown of all 50 sampled pages, near the top of content markdown-availability ✓ markdown-url-support: 50/50 sampled pages support .md URLs (100%) ✓ content-negotiation: 50/50 sampled pages support content negotiation (100%) page-size ✓ rendering-strategy: All 50 sampled pages contain server-rendered content ✗ page-size-markdown: 1 of 50 pages exceed 100K chars (max 314K) Learn more: https://agentdocsspec.com/spec/#page-size-markdown ✗ page-size-html: 1 of 50 sampled pages convert to over 100K chars (max 1319K HTML → 103K markdown (96% boilerplate)) Learn more: https://agentdocsspec.com/spec/#page-size-html ✓ content-start-position: Content starts within first 10% on all 50 sampled pages (median 0%) content-structure ✓ tabbed-content-serialization: No tabbed content detected across 50 sampled pages ✓ section-header-quality: No tabbed content found; header quality check not applicable ✓ markdown-code-fence-validity: All 143 code fences properly closed across 51 pages url-stability ✗ http-status-codes: 50 of 50 sampled pages return 200 for non-existent URLs (soft 404) Learn more: https://agentdocsspec.com/spec/#http-status-codes ✓ redirect-behavior: No redirects detected across 50 sampled pages observability ✗ llms-txt-coverage: llms.txt covers 0/1521 sitemap doc pages (0%); 1521 missing; 16 nested indexes omitted (696 sitemap pages excluded); 702 llms.txt links not in sitemap (may indicate stale links or incomplete sitemap) Learn more: https://agentdocsspec.com/spec/#llms-txt-coverage ✓ markdown-content-parity: All 50 pages have equivalent markdown and HTML content (avg 0% missing) ⚠ cache-header-hygiene: 1 of 51 endpoints have moderate cache lifetimes (1–24 hours) Learn more: https://agentdocsspec.com/spec/#cache-header-hygiene authentication ✓ auth-gate-detection: All 50 sampled pages are publicly accessible ○ auth-alternative-access: All docs pages are publicly accessible; no alternative access paths needed Summary 17 passed, 1 warnings, 4 failed, 1 skipped (23 total) ``` This run shows that the site is reasonably well optimized, but also surfaces some issues with content size and coverage, among others. The output is divided per category as organized in the [agent-friendly docs spec](https://agentdocsspec.com/). > A quick aside: each run by default randomly selects 50 pages listed in the site's llms.txt or sitemap, so results may vary across runs. This is easily adjustable via the [config](https://afdocs.dev/reference/cli), so you can get full-site, deterministic scans to plug into CI/CD workflows as needed, for example. Each flagged issue is followed by a direct link to the corresponding section inside the spec. If we take a look at the first flagged issue - [page-size-markdown](https://agentdocsspec.com/spec/#page-size-markdown) - we get a description of the evaluation metric, pass/fail criteria, and the justification for inclusion and selected thresholds. For the character limit criterion, 50 thousand characters was chosen as the upper threshold due to content "fitting comfortably within all known limits, including Claude Code’s direct-delivery threshold for trusted sites." Other violations, such as the soft-404 flag, or the llms-txt-coverage issues flagged by the tool may depend on your project infrastructure, desired behavior, and framework or tool limitations. This particular project contains thousands of pages and to reduce agent context load splits llms.txt into multiple nested indexes. Additionally, some content in the sitemap is intentionally omitted from _llms.txt_ and vice versa. It also runs under React's PPR/RSC, which is notoriously finicky when it comes to returning true 404 status codes due to how its response preflush works, which trips the soft-404 flag. In short, don't blindly trust all the results the tool throws at you. The only one that knows how and why things work the way they do in your code is yourself. Act only on what you know is a gap in the content or implementation. ## Configuring the tool The demo run used project defaults, but the tool allows us to tweak the behavior. I settled on this after playing around with it a bit: ```yaml url: https://docs.example.com options: # Give deterministic reruns and increase sampled URLs samplingStrategy: deterministic maxLinksToTest: 200 # Set a pass threshold for CI/CD runs coveragePassThreshold: 85 coverageWarnThreshold: 60 # Exclude path globs coverageExclusions: - '**/release-notes/**' ``` Full config reference: . Configs will of course vary depending on your use case. If you want a reproducible config for CI validation you will likely end up with something similar. If you want an ultra strict one-time audit, you will likely end with a zero-tolerance threshold and a full URL audit. ## Conclusion For accuracy, as far as I can tell, each finding is grounded by best practices and research, and backed by verified public sources such as official vendor documentation and other first-party materials. But mainly, the recommendations are sensible and match what any individual experienced with agentic workflows likely already internalized. Take the output as something that helps you guide both the _site_ and _content_ architecture to satisfy human and agentic visitors alike. Caveat being that the landscape shifts rapidly and often. What is considered best practice today may not be seen as such tomorrow. However, as long as the tooling and the spec remain actively maintained, I consider it a reliable signal that helps with decision making. --- Sources: - - - -