…
```
An agent fetching that page receives Tailwind utility classes, responsive grid wrappers, and raw JavaScript functions. It has to reconstruct the actual document content from the props of an unrendered component.
While not as bad as serving raw HTML with no alternative and forcing the LLM to parse through a messy DOM tree, it nevertheless exposes an interesting gap in currently available tooling.
By contrast, fetching Vercel's documentation returns a clean markdown document:
```md
---
title: Vercel Functions
product: vercel
url: /docs/functions
canonical_url: "https://vercel.com/docs/functions"
last_updated: 2026-07-15
type: conceptual
…
---
# Vercel Functions
When you deploy your application, Vercel automatically sets up
the tools and optimizations for your chosen [framework](/docs/frameworks).
It ensures low latency by routing traffic through Vercel's [CDN](/docs/cdn),
and placing your functions in a specific region when you need more control
over [data locality](/docs/functions#functions-and-your-data-source).
```
The takeaway from this should be obvious. Serving agent-ready content requires compiling the source into distinct representations: HTML for humans and markdown for agents.
While this post focuses on `.mdx` sources specifically, the approaches outlined here apply to any structured documentation format.
## HTML converters versus raw MDX
Serving markdown from docs-as-code repositories generally follows one of several strategies:
### Converting HTML
Some platforms run an edge HTML-to-markdown converter (like [Turndown](https://github.com/mixmark-io/turndown) or Mozilla Readability) over the generated HTML page. Cloudflare, for example, offers this at the CDN layer.
While HTML conversion requires zero per-component maintenance, it destroys structured content.
Syntax-highlighted code blocks are the clearest casualty. Modern frameworks wrap highlighted code tokens in nested `
` elements and custom line containers (for per-line highlighting, etc.), and converting that markup back to text frequently corrupts indentation, tabs, and language identifiers. Interactive tabs, accordion groups, and multi-column grids collapse into generic `` hierarchies that lose their contextual relationships.
### Raw source
Serving the raw `.mdx` file directly is the cheapest option, but it offloads compilation artifacts onto the reader:
- Unrendered JSX tags (`
`, ``) clutter the text.
- Build-time dynamic helpers (such as file snippet embedders or automatic cross-reference resolvers) remain unresolved.
- Unused client imports and exported configuration objects waste model context and tokens.
### AST reparse
To generate a stable, most faithful representation, you fork at the MDX source parse. First, generate an abstract syntax tree, transform custom JSX nodes into their semantic markdown equivalents, then stringify to clean markdown. The UnifiedJS ecosystem is very well suited for JSX/MDX based sources.
## The pipeline
Serving agent-ready content from MDX requires decoupling the agent projection from the page compiler. Each document compiles through two independent pipelines: Next.js renders HTML for browsers, while a dedicated [UnifiedJS](https://unifiedjs.com/) pipeline transforms custom tags into clean, semantic markdown for agents.
_One pipeline generates HTML, the other emits clean markdown._
To retain semantic markdown nodes, the markdown pipeline parses the raw file string independently using `remark-parse` and `remark-mdx`:
```ts
/**
* Compiles an MDX source document into self-contained, clean markdown.
*
* Prunes authoring frontmatter, projects custom JSX elements into standard mdast
* structures, normalises paragraph blocks, and absolutises site URLs.
*
* @param source - Raw MDX source content including frontmatter.
* @param origin - Base site origin used to resolve relative links.
* @param options - Projection and diagnostic configuration.
* @returns Serialised markdown string.
* @throws {ProjectionError} When encountering unregistered tags, dynamic expressions, or invalid block nesting.
*/
export function projectMdxToMarkdown(
source: string,
origin: string,
options: ProjectOptions = {},
): string {
const tree = parseMdx(source);
reduceFrontmatter(tree, options.frontmatter);
try {
// Applies per-tag projections.
tree.children = projectChildren(tree.children) as Root["children"];
liftBlocksFromParagraphs(tree);
} catch (error) {
// Surfaces line:column coordinates for authoring errors.
if (error instanceof ProjectionError && options.file) {
throw atFile(options.file, error);
}
throw error;
}
rewriteUrls(tree, origin);
// Turns the augmented mdast into a markdown document.
return unified()
.use(remarkStringify, {
bullet: "-",
emphasis: "_",
fences: true,
listItemIndent: "one",
rule: "-",
})
.use(remarkGfm)
.use(remarkFrontmatter, ["yaml"])
.stringify(tree);
}
```
Alternatively, visualized by a flowchart:
Let's walk through the implementation step-by-step.
### Read the file
The projection starts directly from disk by reading the raw source string with no bundler or module evaluation.
```ts
/**
* Projects an MDX content source file into clean markdown.
*
* Evaluates custom tags, normalises links against the site origin, and appends
* canonical metadata.
*
* @param urlPath - Content route path without a leading slash.
* @returns The projected markdown content string.
* @throws {Error} When the source file cannot be read or AST projection fails.
* @cached self
*/
export async function projectPageSource(urlPath: string): Promise {
"use cache";
cacheLife("max");
cacheTag(pageCacheTag(urlPath));
const source = await readFile(
path.join(CONTENT_DIR, ...urlPath.split("/")) + ".mdx",
"utf8",
);
return projectMdxToMarkdown(source, SITE_ORIGIN, {
file: `content/${urlPath}.mdx`,
frontmatter: { url: `${SITE_ORIGIN}/${urlPath}`, author: AUTHOR.name },
});
}
```
The extra arguments are used as metadata during evaluation:
- `origin` - site domain used when generating absolute URLs
- `file` - used to set line:column coordinates for surfaced errors
- `frontmatter` - additional properties merged with the loaded file's frontmatter
### Parse
Next, the string gets parsed by the `unified()` pipeline.
```ts
/**
* Parses MDX source into mdast with the syntax extensions the site's content
* uses: GFM, a YAML frontmatter block, and JSX.
*
* @param source - Full file contents, frontmatter included.
* @returns The tree; frontmatter is a `yaml` node, JSX is `mdxJsx*` nodes.
* @throws {Error} When the source is not valid MDX; the message carries the
* line and column.
*/
export function parseMdx(source: string): Root {
return unified()
.use(remarkParse)
.use(remarkGfm)
.use(remarkFrontmatter, ["yaml"])
.use(remarkMdx)
.parse(source) as Root;
}
```
Which outputs a parsed syntax tree. For example, the input:
```mdx
Status: ok today.
Mind the gap.
```
produces the following tree.
**Parsed AST tree**
```json
{
"type": "root",
"children": [
{
"type": "paragraph",
"children": [
{ "type": "text", "value": "Status: " },
{
"type": "mdxJsxTextElement",
"name": "Ink",
"attributes": [
{ "type": "mdxJsxAttribute", "name": "c", "value": "green" }
],
"children": [{ "type": "text", "value": "ok" }]
},
{ "type": "text", "value": " today." }
]
},
{
"type": "paragraph",
"children": [
{
"type": "mdxJsxTextElement",
"name": "Note",
"attributes": [
{ "type": "mdxJsxAttribute", "name": "title", "value": "Careful" }
],
"children": [{ "type": "text", "value": "Mind the gap." }]
}
]
}
]
}
```
The plugins handle various cases:
- `remark-frontmatter` parses the leading YAML block as a `yaml` node.
- `remark-mdx` retains JSX tags as structured AST nodes.
- `remark-gfm` (GitHub-flavored markdown) adds support for additional syntax such as tables and autolinks.
#### Node types and edge cases
Consider an inline tag versus a standalone tag:
```mdx
Status: ok today.
Mind the gap.
```
As seen in the snippet above, both parse as `mdxJsxTextElement` inside a parent `paragraph`.
MDX emits a `mdxJsxFlowElement` only when opening, children, and closing tags each occupy separate lines. Single-line tags always parse as phrasing (inline) content inside paragraphs:
- `Mind the gap.` on one line -> `paragraph > mdxJsxTextElement`
- `` and `` on separate lines -> `mdxJsxFlowElement`
- `Mind the gap.` inline -> `paragraph > mdxJsxTextElement`
- `` on one line -> `mdxJsxFlowElement`
Handling both node types forces us to extract single-line nodes such as `` to a separate paragraph via an additional post-processing pass. Otherwise, we end up with the element's output projected inline for these cases.
### Modify the tree
With the parsed mdast in memory, `projectChildren` traverses the tree depth-first while `projectNode` converts each node type to its target markdown representation.
| Node Type | Input Pattern | Transformation |
| --------------------------- | -------------------------------- | ----------------------------------------- |
| MDX constructs (`mdxjsEsm`) | `import`, `export`, `{ js }` | Dropped (`null`) |
| Standard mdast | Headings, lists, code fences | Passes through; children walked in place |
| Declared custom tag | Tag registered in `tag-names.ts` | Replaced by bound projection return value |
| Undeclared capitalised tag | `` | Throws `ProjectionError` (fails build) |
| Lowercase HTML element | ``, `
` | Unwrapped; children replace container |
`projectNode` maps each mdast node to its replacement:
```ts
/**
* Transforms a single AST node and its descendants into markdown nodes.
*
* Passes standard mdast through, evaluates registered custom JSX tags, drops
* MDX expressions and ESM blocks, and unwraps lowercase HTML elements.
*
* @param node - AST node to project.
* @returns Replacement mdast node(s), or null to discard the node.
* @throws {ProjectionError} When encountering an unregistered capitalised tag or when a component projection throws.
*/
function projectNode(node: RootContent): ProjectionResult {
if (
node.type === "mdxjsEsm" ||
node.type === "mdxFlowExpression" ||
node.type === "mdxTextExpression"
) {
return null;
}
if (node.type !== "mdxJsxFlowElement" && node.type !== "mdxJsxTextElement") {
if ("children" in node) {
const parent = node as Parents;
parent.children = projectChildren(
parent.children as RootContent[],
) as Parents["children"];
}
return node;
}
const name = node.name ?? "";
if (isTagName(name)) {
let result: ProjectionResult;
try {
result = projections[name](node, projectChildren);
} catch (error) {
throw locate(error, node);
}
// Skips block elements.
if (node.type !== "mdxJsxFlowElement" || result == null) return result;
// Ensures a block element for orphaned inline (phrasing) elements.
return groupIntoBlocks(Array.isArray(result) ? result : [result]);
}
if (isCapitalised(name)) {
throw new ProjectionError(
`markdown-projection: <${name}> is not a tag this site declares. ` +
`A tag is declared in lib/projection/tag-names.ts and bound in mdx-components.tsx and lib/projection/tags.ts.`,
node.position,
);
}
return projectChildren(node.children as RootContent[]);
}
```
### Serialize
Finally, serialize the tree to clean markdown with another `unified` instance:
```ts
// Turns the augmented mdast into a markdown document.
return unified()
.use(remarkStringify, {
bullet: "-",
emphasis: "_",
fences: true,
listItemIndent: "one",
rule: "-",
})
.use(remarkGfm)
.use(remarkFrontmatter, ["yaml"])
.stringify(tree);
```
## Component projections
A component projection is a pure function that transforms a JSX AST node into standard mdast nodes:
```ts
/**
* Result of projecting a JSX element: transformed mdast node(s), or null to drop.
*/
export type ProjectionResult = RootContent | RootContent[] | null;
/**
* Transforms an MDX JSX element into standard mdast markdown nodes.
*
* @param node - The JSX element node with its attributes and unprojected children.
* @param projectChildren - Recursive projection function for child node lists.
* @returns Transformed mdast node(s), or null to drop the element.
*/
export type Projection = (
node: MdxJsxElement,
projectChildren: (nodes: RootContent[]) => RootContent[],
) => ProjectionResult;
```
Binding a projection to every tag declared across your site turns "what does this component mean in markdown?" into a concrete type contract. Answering that question is where the actual work lives, and the implementation depends entirely on what the original component was designed to do.
### Unwrap and drop
Tags that exist purely to colour, wrap, or position their children carry no structural meaning for an agent. In those cases, the projection returns the projected children directly and the outer wrapper drops away:
```mdx
Status: ok today.
```
```md
Status: ok today.
```
Column sets, responsive grids, and card decks follow the exact same pattern. Elements that exist purely for human interactivity such as code block copy buttons resolve to nothing. Audience-conditional blocks resolve according to which branch is allowed to survive:
```mdx
Only people read this.
Only agents read this.
```
```md
Only agents read this.
```
Both can be defined as shared helpers in your tag registry.
```ts
/** Drops the element from markdown output entirely. */
export const dropFromMarkdown: Projection = async () => null;
/** Unwraps children inline. */
export const unwrapInMarkdown: Projection = async (node) =>
node.children as RootContent[];
```
### Structural tags
Markdown already provides native constructs for the components used in most documentation styleguides. A callout banner maps naturally to a blockquote prefixed with a bold label:
```mdx
Mind the gap.
Mind the gap.
```
```md
> **Note: Careful**
>
> Mind the gap.
> **Note:** Mind the gap.
```
When a tag can appear in both block and inline contexts, its projection should adapt accordingly. An inline tag sitting within a sentence produces an `inlineCode` node, whereas that same tag standing on its own line receives blockquote formatting. Use the AST node's own type (`mdxJsxTextElement` vs. `mdxJsxFlowElement`) to choose the right output.
### Hidden content
Disclosure widgets, tab sets, and accordions show only one piece of content at a time in HTML. Markdown has no client-side state or click handlers to toggle visibility; preserving that hidden state in a projection simply throws content away that a human reader can readily reach.
Instead, the projection expands every branch beneath a descriptive heading:
```mdx
Some _body_ text, then a list:
- one item
- another
```
```md
**Build output**
Some _body_ text, then a list:
- one item
- another
```
Tab sets follow the same principle: the container unwraps its children back through the AST walker, emitting each tab's contents beneath its corresponding label.
### Build helpers
Unresolved dynamic helpers are one of the primary flaws of serving raw MDX to LLMs. Because the projection pipeline runs during the static build with direct access to the filesystem and site metadata, it can resolve dynamic helpers into literal content.
Every code plate in this post uses a `` tag: it references a source file path and an optional region ID, and the projection reads the file off disk to emit a syntax-highlighted code fence. Similarly, an internal `` resolves the target document's title and appends a `.md` suffix to the URL, converting an empty tag into an actionable link:
```mdx
```
```md
[Customizing the Claude Code statusline](https://example.com/blog/cc-statusline.md)
```
Because projections operate as build-time AST transforms rather than simple string replacements, broken file paths or dead internal links can fail the static build immediately before reaching production.
## Block lifting
In MDX, authors frequently place custom component tags inline within regular paragraphs:
```mdx
Check out our utility configuration: for origin handling.
```
When `` transforms into a fenced code block, having a block-level node nested inside a `paragraph` AST node produces invalid parse. Standard serializers like `remark-stringify` will emit the code block without the mandatory surrounding blank lines, leading to corrupted outputs.
To keep the AST valid, a lifting pass traverses the tree after component transformation and splits parent paragraphs around block nodes:
_The lifting pass splits parent paragraphs around block nodes to preserve mdast hierarchy._
The lifting pass ensures every block-level node sits at the top level of the AST:
```ts
/**
* Recursively extracts block-level nodes out of enclosing paragraph nodes.
*
* Splits parent paragraphs around block nodes so serializers emit proper
* blank line separators around fences, blockquotes, and lists.
*
* @param parent - AST parent node whose subtree is rewritten in place.
* @throws {ProjectionError} When a block node resides within an unliftable phrasing container such as a table cell.
*/
export function liftBlocksFromParagraphs(parent: Parents): void {
if (parent.type === "tableCell") {
for (const child of parent.children as RootContent[]) {
if (BLOCK_TYPES.has(child.type) && child.type !== "paragraph") {
throw new ProjectionError(
`markdown-projection: a <${child.type}> block cannot be projected inside a table cell.`,
parent.position,
);
}
}
}
const out: RootContent[] = [];
for (const child of parent.children as RootContent[]) {
if (child.type !== "paragraph") {
if ("children" in child) liftBlocksFromParagraphs(child as Parents);
out.push(child);
continue;
}
/**
* Emits a phrasing run as a paragraph node, discarding empty whitespace runs.
*
* @param group - Accumulated phrasing nodes.
*/
const pushPhrasing = (group: PhrasingContent[]): void => {
const trimmed = trimPhrasing(group);
if (trimmed) {
out.push({ type: "paragraph", children: trimmed } satisfies Paragraph);
}
};
const phrasing: PhrasingContent[] = [];
for (const sub of child.children) {
if (!BLOCK_TYPES.has(sub.type)) {
phrasing.push(sub);
continue;
}
pushPhrasing(phrasing.splice(0));
if ("children" in sub) liftBlocksFromParagraphs(sub as Parents);
out.push(sub as RootContent);
}
pushPhrasing(phrasing);
}
(parent as { children: RootContent[] }).children = out;
}
```
The inverse can occur as well. A tag written alone on its own line occupies a block slot in the MDX AST, but its projection might produce inline phrasing content. For instance, `` and `` each return a single `link` node.
```ts
/**
* Projects a downloadable file reference into a markdown hyperlink.
*
* @param node - The Download JSX element AST node.
* @returns Link AST node pointing to the target asset URL.
* @throws {Error} When `src` attribute is missing or non-literal.
*/
Download: (node) => {
const src = getAttr(node, "src");
if (!src) {
throw new Error(
"markdown-projection: requires a literal src attribute.",
);
}
// Returns a mdast link node.
return {
type: "link",
url: src,
children: [
{ type: "text", value: getAttr(node, "label") ?? publicFileName(src) },
],
};
},
```
If emitted into a block slot without a parent paragraph, the converter prepends to adjacent paragraphs without any separation or whitespace.
```ts
// Skips block elements.
if (node.type !== "mdxJsxFlowElement" || result == null) return result;
// Ensures a block element for orphaned inline (phrasing) elements.
return groupIntoBlocks(Array.isArray(result) ? result : [result]);
```
---
Sources:
- UnifiedJS —
- IMG.LY, "Make Your Docs Agent-Ready: MDX to Markdown" —
- Fumadocs, "Remark LLMs" and "Markdown Rendering" —
- Cloudflare, "Markdown for Agents" —
- mdast syntax —
…