---
title: Routing agentic traffic
description: Route incoming requests between HTML and markdown based on context.
date: 2026-08-30
tags:
  - ai
  - documentation
url: https://skurekjakub.dev/blog/audience-aware-routing
author: Jakub Skurek
---

> Agent instructions:
> Take page paths from the index rather than constructing them — guessed paths 404. Links inside this document already point at `.md`; following them, or sending `Accept: text/markdown`, keeps you in Markdown.
> When citing this page, use the `url` in its frontmatter — the HTML page; this `.md` address is a rendering of it.
>
> - [llms.txt](https://skurekjakub.dev/llms.txt) — every page with a Markdown twin, with descriptions.
> - [skill.md](https://skurekjakub.dev/skill.md) — how to read this site as an agent.

AI agents strongly prefer markdown over HTML. It is token-efficient, eliminates the overhead of parsing DOM trees and stripping scripts, and works natively across modern coding tools like Claude Code, Codex, Copilot, and Cursor.

As websites and documentation platforms add machine-readable markdown representations alongside traditional HTML pages, they run into an immediate architectural question: when a request comes in, how do you decide which format to serve?

In HTTP, there are three primary signals to route on:

- The explicit URL path (`/blog/foo.md`)
- The `Accept` header (`Accept: text/markdown`)
- The `User-Agent` string (`Claude-User`, `cursor/`)

Each signal operates at a different level of reliability and carries distinct caching implications.

In this post, we will look at how to structure routing between HTML and markdown representations, how to handle content negotiation without breaking shared CDN caches, and how to handle pages that exist only in HTML.

## Explicit `.md` URLs

The simplest and most reliable approach is exposing explicit URLs ending in `.md`. When a client requests `/blog/foo.md`, the server returns the markdown representation directly.

```text
GET /blog/foo.md        → 200 text/markdown   (the projection)
GET /blog/foo           → 200 text/html       (the page)
```

Try it on this blog:

- [Markdown — /blog/audience-aware-routing.md](https://skurekjakub.dev/blog/audience-aware-routing.md)
- [HTML — /blog/audience-aware-routing](https://skurekjakub.dev/blog/audience-aware-routing.md)

Explicit URLs have two major architectural advantages:

- **They are unconditional:** If a user or crawler follows a `.md` link in a desktop browser sending `Accept: text/html`, the server still delivers markdown. The URL path takes precedence over incoming headers, preventing client-side header misconfigurations from altering the response.
- **They own their cache key:** Because `/blog/foo.md` and `/blog/foo` are separate URL paths, your CDN and browser cache them as independent static resources without any `Vary` header coordination.

In an edge proxy or middleware, a `.md` suffix can short-circuit negotiation before inspecting any other request headers. Rewriting the request internally onto a projection route keeps the clean public URL as the CDN cache key:

```ts
/**
 * Routes requests to Markdown projections based on URL or negotiated headers.
 *
 * Explicit `.md` URLs route unconditionally to Markdown projections. Other URLs
 * route to Markdown when headers select it via {@link selectsMarkdown}. Non-GET
 * and non-HEAD requests bypass negotiation.
 *
 * @param request - Incoming HTTP request.
 * @param url - Parsed request URL.
 * @returns The rewritten Markdown response, a Markdown 404, or null to continue
 *   the proxy chain.
 */
export const negotiateMarkdownProjection: ProxyStep = async (request, url) => {
  if (request.method !== "GET" && request.method !== "HEAD") return null;
  const byUrl = url.pathname.endsWith(".md");
  if (!byUrl && !selectsMarkdown(request)) return null;
  return respondInMarkdown(markdownAnswerFor(url.pathname), url, !byUrl);
};
```

To make explicit URLs discoverable to agents, an [`/llms.txt`](https://skurekjakub.dev/llms.txt) index at the domain root can enumerate every available markdown page.

Additionally, HTML pages can advertise their markdown counterparts via a `Link` HTTP header and a `<link rel="alternate" type="text/markdown">` tag:

```http
link: </blog/foo.md>; rel="alternate"; type="text/markdown", </llms.txt>; rel="llms-txt", </llms-full.txt>; rel="llms-full-txt", </.well-known/agent-skills/index.json>; rel="agent-skills"
```

While not every AI tool parses `Link` headers today, advertising it provides a clean discovery mechanism for link-aware agents at zero operational cost.

## Content negotiation with `Accept`

The standard HTTP mechanism for serving different representations of the same resource is [content negotiation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Content_negotiation). Because `text/markdown` is a registered media type ([RFC 7763](https://www.rfc-editor.org/rfc/rfc7763)), a client can state its preference via the `Accept` [header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept).

When an agent requests `/blog/foo` with `Accept: text/markdown`, the server returns the markdown representation. Regular browsers, which send `Accept: text/html,...`, continue receiving standard HTML.

The negotiation parser parses the header, extracts quality values (`q`), and selects markdown only when it outranks HTML:

```ts
/** Media type entry parsed from an `Accept` header. */
interface AcceptEntry {
  /** Quality value in `[0, 1]`. */
  q: number;
  /** Position in the `Accept` header list, used to break quality-value ties. */
  index: number;
}

/**
 * Locates a media type in an `Accept` header.
 *
 * Missing or unparseable `q` parameters default to 1 (RFC 9110 § 12.4.2).
 *
 * @param accept - Lowercased `Accept` header value.
 * @param mediaType - Exact media type to match.
 * @returns The matching entry, or null when absent.
 */
function findAcceptEntry(
  accept: string,
  mediaType: string,
): AcceptEntry | null {
  const entries = accept.split(",");
  for (const [index, entry] of entries.entries()) {
    const [type, ...params] = entry.split(";").map((part) => part.trim());
    if (type !== mediaType) continue;

    const q = Number.parseFloat(
      params.find((p) => p.startsWith("q="))?.slice(2) ?? "1",
    );
    return { q: Number.isNaN(q) ? 1 : q, index };
  }
  return null;
}

/**
 * Evaluates whether request headers select Markdown representation.
 *
 * Checks `Accept` header preference between `text/markdown` and `text/html`,
 * falling back to `User-Agent` detection when neither type is specified.
 *
 * @param request - Incoming HTTP request.
 * @returns True when `Accept` selects Markdown or when `User-Agent` matches
 *   and media types are omitted; false when HTML is preferred or Markdown is refused.
 */
function selectsMarkdown(request: NextRequest): boolean {
  const accept = (request.headers.get("accept") ?? "").toLowerCase();
  const markdown = findAcceptEntry(accept, "text/markdown");
  const html = findAcceptEntry(accept, "text/html");

  // Explicit q=0 indicates "not acceptable" (RFC 9110 § 12.5.1) and must outrank
  // the User-Agent fallback.
  if (markdown !== null && markdown.q <= 0) return false;

  if (markdown !== null) {
    if (html === null || html.q <= 0) return true;
    if (markdown.q > html.q) return true;
    // Equal q-values break ties by header order (RFC 9110 § 12.5.1).
    if (markdown.q === html.q && markdown.index < html.index) return true;
  }

  // Explicit Accept for HTML takes precedence over User-Agent detection.
  if (html !== null && html.q > 0) return false;

  return prefersMarkdownByUserAgent(request.headers.get("user-agent") ?? "");
}
```

Ties in quality values resolve based on header order (`text/markdown,text/html` prefers markdown, whereas `text/html,text/markdown` prefers HTML).

If a client sends `text/markdown;q=0`, it is explicitly opting out of markdown. That refusal overrides all other heuristics—including `User-Agent` sniffing. In that case, only an explicit `.md` URL will return markdown.

Here is an example request header sent by Claude Code v2.1.251:

```text
Accept: text/markdown, text/html, */*
Accept-Encoding: gzip, compress, deflate, br
User-Agent: Claude-User (claude-code/2.1.251; +https://support.anthropic.com/)
Host: skurekjakub.dev
```

### Cache handling

In theory, HTTP already has a mechanism for serving different representations from the same URL: the `Vary` [header](https://httpwg.org/specs/rfc9110.html#field.vary). When an agent requests markdown at `/blog/foo`, the server tells downstream caches that the response depends on the incoming `Accept` header by returning `Vary: Accept`.

In practice, relying on `Vary` across modern CDNs and web frameworks runs into several messy edge cases:

- **`User-Agent` fragmentation:** When an agent sends `Accept: */*` or omits the header entirely, routing falls back to `User-Agent`. But you can't safely put `User-Agent` in `Vary` without destroying cache hit rates—browser strings carry thousands of permutations of versions, OS builds, and architectures. If you only vary on `Accept`, a bot's markdown response gets cached under `*/*` and served to browsers.
- **CDN support:** While origin proxies (like nginx or Varnish) respect `Vary`, many CDNs ignore `Vary` headers other than `Accept-Encoding` unless explicitly configured with custom cache keys. An unconfigured edge cache that ignores `Vary` simply stores whichever response arrived first, serving raw markdown to human visitors or HTML to agents.
- **Framework ownership:** In Next.js App Router, `Vary` is owned by the React Server Components wire protocol (`rsc`, `next-router-state-tree`, etc.) to separate component payloads from full HTML pages. The framework manages the header downstream, so custom values added in middleware or config get stripped or overwritten.

Because of these trade-offs, modern architectures (including Next.js itself) increasingly avoid `Vary` and move toward pathname-based caching instead.

The practical fix is separating what you cache from what you negotiate:

- **Explicit `.md` URLs for caching:** The `.md` URL (`/blog/foo.md`) is its own path. CDNs and browsers cache it like any static asset without needing `Vary` configuration.
- **`private, no-store` for negotiation:** Negotiated requests on canonical URLs (`/blog/foo`) opt out of shared caching with `cache-control: private, no-store`. This serves dynamic markdown to agents on demand without any risk of polluting the CDN cache for regular traffic.

Here is what the response headers look like in practice on this site:

```text
$ curl -sI https://skurekjakub.dev/blog/cc-statusline.md
HTTP/2 200
cache-control: public, max-age=0, must-revalidate
content-type: text/markdown; charset=utf-8
x-vercel-cache: HIT

$ curl -sI -H 'Accept: text/markdown' https://skurekjakub.dev/blog/cc-statusline
HTTP/2 200
cache-control: private, no-store
content-type: text/markdown; charset=utf-8
vary: rsc, next-router-state-tree, next-router-prefetch, next-router-segment-prefetch
```

The `.md` URL's `max-age=0, must-revalidate` lets the CDN hold a shared cache entry (reported by `x-vercel-cache: HIT`) while browsers revalidate. Meanwhile, the negotiated response safely serves on-demand markdown with `private, no-store`—carrying the framework's internal RSC `vary` headers without risking CDN cache bleed.

## User-Agent detection

When incoming requests lack specific media type headers (for instance, an agent sending `Accept: */*`), the only remaining signal is the `User-Agent` string. The proxy maintains a list of known agent tokens:

```ts
/** User-Agent substrings for agents preferring Markdown representation. */
const AGENT_USER_AGENT_TOKENS: readonly string[] = [
  "anthropic-ai",
  "amzn-user",
  "bytespider",
  "ccbot",
  "chatgpt-user",
  "claude-code",
  "claude-user",
  "claude-web",
  "claudebot",
  "cohere-ai",
  "duckassistbot",
  "gemini-deep-research",
  "google-gemini-cli",
  "google-notebooklm",
  "googleagent-urlcontext",
  "gptbot",
  "kagi-fetcher",
  "kimi-user",
  "manus-user",
  "meta-externalagent",
  "meta-externalfetcher",
  "mistralai-user",
  "opencode",
  "perplexity-user",
];

/**
 * Evaluates whether a User-Agent header matches known AI or agent clients.
 *
 * @param userAgent - Raw `User-Agent` header value.
 * @returns True when the value contains one of {@link AGENT_USER_AGENT_TOKENS}, matched case-insensitive.
 */
export function prefersMarkdownByUserAgent(userAgent: string): boolean {
  const lowered = userAgent.toLowerCase();
  return AGENT_USER_AGENT_TOKENS.some((token) => lowered.includes(token));
}
```

However, User-Agent detection requires careful hygiene to avoid breaking search engine crawlers that **need** HTML to extract JSON-LD metadata, OpenGraph tags, and page links.

For example, matching a broad substring like `claude` will inadvertently catch `Claude-SearchBot`, just as `perplexity` will match `PerplexityBot`, and `chatgpt` will match the ChatGPT Atlas browser. Search crawlers index your site for human search engines and must continue receiving full HTML.

### Maintaining the agent list

Agent user-agent strings evolve rapidly. There are two common approaches to list maintenance:

- **Manual curation:** Maintain a focused list of known coding assistants and scrapers directly in code.
- **Automated sync:** Community projects like [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) publish structured `robots.json` files categorizing bots by function (AI Assistants, Search Crawlers, Scrapers).

A simple GitHub Actions workflow can automate upstream list updates:

```yaml
on:
  schedule: [{ cron: '0 4 * * *' }]
jobs:
  sync:
    steps:
      - fetch robots.json from upstream
      - filter entries categorized as AI assistants or coding agents
      - regenerate agent-markdown-user-agents.ts
      - open a PR if changes are detected
```

## Routing precedence

Combining these signals yields a deterministic evaluation order:

```text
1. .md URL                     → markdown, unconditionally
2. Accept: text/markdown;q=0   → HTML (explicit opt-out)
3. Accept ranks md above html  → markdown
4. Accept: text/html           → HTML
5. User-Agent on agent list    → markdown (fallback guess)
6. otherwise                   → HTML
```

![Routing priority for markdown responses.](https://skurekjakub.dev/blog/audience-aware-content/routing-precedence.drawio.svg)

Once a request determines that markdown is desired, the next step is checking whether the target route supports a markdown representation.

## Handling routes without markdown

Determining that a request wants markdown is not the end of the journey. Next, we need to generate the correct response based on the requested route. In general, we can classify a few route types:

- Blog posts (`/blog/[slug]`), documentation articles, and general prose content.
- Index-like pages such as `/` and `/blog` that contain dynamically composed content aggregated from other sources.
- Pages that should not return markdown regardless: interactive tools, contact forms, or dynamic dashboard listings.
- Assets like `robots.txt`, sitemaps, or Open Graph images.
- Invalid paths (404s) that don't exist in the application.

This implies a routing matrix that needs to be evaluated at the proxy layer:

![Routing decision tree mapping request paths and entry mechanisms to their corresponding markdown or HTML responses.](https://skurekjakub.dev/blog/audience-aware-content/markdown-answer.drawio.svg)

The critical edge case is when an agent requests markdown for a route that only exists in HTML, such as `/contact`.

The first obvious intuition could be to return a markdown-formatted 404:

```http
HTTP/2 404 Not Found
content-type: text/markdown; charset=utf-8

# Not found

This page is not available in markdown: `/contact`.
The index of every markdown page: https://skurekjakub.dev/llms.txt
```

In practice, returning a 404 for an existing HTML page creates more issues:

- When an LLM tool (such as Claude Code's `WebFetch`) encounters an HTTP 404, it aborts and the model only ever sees `"The server returned HTTP 404 Not Found. The response body was not retrieved."`
- Real agent clients (Claude Code, Cursor, Copilot) send `Accept: text/markdown, text/html, */*`, indicating that HTML is an acceptable fallback.
- Bots like `GPTBot` or `ClaudeBot` treat a 404 as a removed page and purge the URL from their index.

Same as with caching, the solution is to split the behavior on the requested path:

- Explicit `.md` URLs, like `/nothing-here.md`, return `404 text/markdown`. If the request targets something that is not there under any circumstances, a Not Found is a valid response.
- Negotiated requests (`Accept` or `User-Agent` on `/nothing-here`) fall through to standard HTML routing (`200 text/html`) with the discovery `Link` header attached.

```text
$ curl -si -A 'Claude-User' https://skurekjakub.dev/nothing-here
HTTP/2 200
content-type: text/html; charset=utf-8
link: </llms.txt>; rel="llms-txt", </llms-full.txt>; rel="llms-full-txt", </.well-known/agent-skills/index.json>; rel="agent-skills"

$ curl -si https://skurekjakub.dev/nothing-here.md
HTTP/2 404
content-type: text/markdown; charset=utf-8
x-robots-tag: noindex

# Not found

This page is not available in Markdown: `/nothing-here`.

The index of every markdown page: https://skurekjakub.dev/llms.txt
```

## Encode markdown support at the route level

The routing layer needs to know if a specific URL has the capacity to respond in markdown.

There are many solutions to this problem, so we'll only look at one. Couple markdown support declaration with the route definition itself. The route declares supported outputs and an intermediate artifact gives the proxy layer enough information to make the routing decision.

On this blog, each `page.tsx` exports its declaration:

```tsx
export const markdown = "twin" satisfies MarkdownDeclaration;
```

Which gets collected and converted to a route map by a simple script:

```ts
// Generated by scripts/generate-route-markdown.ts from app/ — do not edit.
import type { RouteMarkdown } from "./route-markdown";

export const ROUTE_MARKDOWN: readonly RouteMarkdown[] = [
  { route: "/", markdown: "index" },
  { route: "/.well-known/agent-skills/index.json", markdown: "own" },
  { route: "/about", markdown: "twin" },
  { route: "/api/mcp", markdown: "own" },
  { route: "/blog", markdown: "index" },
  { route: "/blog/[slug]", markdown: "twin" },
  { route: "/blog/[slug]/opengraph-image", markdown: "own" },
  { route: "/blog/[slug]/opengraph-image/[id]", markdown: "own" },
  { route: "/blog/tags", markdown: "own" },
  { route: "/blog/tags/[tag]", markdown: "none" },
  { route: "/contact", markdown: "none" },
  { route: "/feed.xml", markdown: "own" },
  { route: "/llms-full.txt", markdown: "own" },
  { route: "/llms.txt", markdown: "own" },
  { route: "/md/[...page]", markdown: "own" },
  { route: "/opengraph-image", markdown: "own" },
  { route: "/robots.txt", markdown: "own" },
  { route: "/sitemap.xml", markdown: "own" },
  { route: "/skill.md", markdown: "own" },
];
```

The declaration contract is defined as a simple union type:

```ts
/**
 * Markdown projection capability declared by a page route.
 *
 * Exported from `app/**\/page.tsx` as
 * `export const markdown = "…" satisfies MarkdownDeclaration`.
 *
 * - `twin`: The route has a markdown projection served from `/md/<path>`.
 * - `index`: The `/llms.txt` corpus index substitutes for the route.
 * - `none`: The route has no markdown projection; negotiated misses fall
 *   through to HTML, and `.md` URL requests receive a markdown 404.
 */
export type MarkdownDeclaration = "twin" | "index" | "none";
```

For example, `app/blog/[slug]/page.tsx` declares its projection twin:

```tsx
export const markdown = "twin" satisfies MarkdownDeclaration;
```

This convention is enforced by a simple CI check that guards against drift when adding new routes.

At runtime, evaluating how to respond is a table lookup:

```ts
/**
 * Resolution target for an incoming markdown request.
 */
export type MarkdownAnswer =
  /** The page's markdown projection, served internally from `/md<pagePath>`. */
  | { kind: "twin"; pagePath: string }
  /** The `/llms.txt` corpus index. */
  | { kind: "index" }
  /** No markdown representation exists for this path. */
  | { kind: "missing"; pagePath: string }
  /** A dedicated handler serves this URL; content negotiation does not intercept. */
  | { kind: "own-route" };

/**
 * Resolves the markdown representation for a requested pathname.
 *
 * Evaluates declared route mappings for standard pages and explicit `.md` URL paths.
 *
 * @param pathname - Request path with leading slash.
 * @returns The resolved {@link MarkdownAnswer}.
 */
export function markdownAnswerFor(pathname: string): MarkdownAnswer {
  let pagePath = pathname;
  if (pathname.endsWith(".md")) {
    if (matchRoute(pathname)?.markdown === "own") return { kind: "own-route" };
    pagePath = pathname.slice(0, -3) || "/";
  }
  switch (matchRoute(pagePath)?.markdown) {
    case "twin":
      return { kind: "twin", pagePath };
    case "index":
      return { kind: "index" };
    case "own":
      return pagePath === pathname
        ? { kind: "own-route" }
        : { kind: "missing", pagePath };
    default:
      return { kind: "missing", pagePath };
  }
}
```

A few things to note:

- Content negotiation applies only to `GET` and `HEAD` requests. Mutation requests like form POSTs pass directly through to their handlers.
- Direct requests to `/blog.md` or `/.md` rewrite immediately to `/llms.txt`.

## Audience-aware content

Even when markdown projections mirror their HTML counterparts, human readers and AI agents often benefit from different instructions for the same underlying task.

For example, "Open **Settings** in the top right and select ☀" is great visual guidance for a human reader. For an agent editing configuration files or executing CLI commands, a direct instruction like "Set `theme` to `'dark'` in `localStorage`" is far more actionable and saves tokens.

In MDX, this can be expressed with a custom `<Visibility>` component:

```mdx
<Visibility for="humans">
Click **Settings** in the top-right and toggle the switch.
</Visibility>
<Visibility for="agents">
Set `theme` to `"dark"` in `localStorage` under the key `theme`.
</Visibility>
```

The `humans` branch renders in HTML and is stripped from the markdown projection. The `agents` branch ships inside HTML in a hidden, machine-readable block so raw parsers can find it, while the markdown projection unwraps it directly. Headings are disallowed inside `<Visibility>` blocks to keep table-of-contents anchors consistent between both renderings.

For more details on building the underlying MDX-to-markdown projection pipeline, check out [this deep dive](https://skurekjakub.dev/blog/serving-markdown-content.md).

## Other options

While HTTP routing and content negotiation work well for public web documentation, other architectural patterns can be considered based on architecture and requirements:

### Model Context Protocol (MCP)

For multi-step agent workflows, exposing a dedicated Model Context Protocol server lets agents search and query documentation over structured JSON-RPC channels rather than scraping web pages. While this post focuses on handling incoming HTTP traffic, MCP provides a powerful alternative for tooling-native interactions.

### Direct filesystem access

For local development and codebase-specific documentation, skip HTTP entirely and point the agent directly at the markdown source files on disk.

## Key points

Building clean routing for agentic traffic comes down to a few core principles:

- Expose explicit `.md` URLs as the primary, publicly cacheable tier.
- Support `Accept` negotiation for smart fallbacks, but protect shared caches with `private, no-store`.
- Fall through to HTML on negotiated misses rather than serving a 404 that agent tools discard.

---

Sources:

- acceptmarkdown.com - <https://acceptmarkdown.com/>
- RFC 9110, HTTP Semantics - <https://httpwg.org/specs/rfc9110.html>
- RFC 7763, the text/markdown media type - <https://www.rfc-editor.org/rfc/rfc7763>
- Next.js proxy file convention - <https://nextjs.org/docs/app/api-reference/file-conventions/proxy>
- Vercel Routing Middleware - <https://vercel.com/docs/routing-middleware>
- llms.txt - <https://llmstxt.org/>
- ai.robots.txt - <https://github.com/ai-robots-txt/ai.robots.txt>