---
title: LLM files
description: >-
  Generate llms.txt for hosted websites and AGENTS.md for npm-bundled offline
  reading.
group: reference
lastModified: '2026-05-11T20:02:32-07:00'
lastAuthor: 'github-actions[bot]'
---
# LLM files

The `leadtype/llm` entry point produces four flavors of agent-facing output, all derived from the same docs source:

* **`generateLlmsTxt`** — for hosted websites. Emits the `/llms.txt` convention with root-relative markdown mirror links.
* **`generateLLMFullContextFiles`** — root `/llms-full.txt` fallback containing all generated markdown docs. Pairs with `generateLlmsTxt`.
* **`generateAgentReadabilityArtifacts`** — docs-scoped `sitemap.xml`, `sitemap.md`, `robots.txt`, and JSON manifest data that a host app can merge into site-level files.
* **`generateAgentsMd`** — for npm-bundled docs. Emits an `AGENTS.md` index with relative `./docs/<topic>.md` paths so the file works inside `node_modules/<pkg>/`.

```ts
import {
  extractDocsTableOfContents,
  generateAgentReadabilityArtifacts,
  generateAgentsMd,
  generateLLMFullContextFiles,
  generateLlmsTxt,
  renderRobotsTxt,
  renderSitemapMarkdown,
  renderSitemapXml,
  resolveDocsNavigation,
  resolveDocsTableOfContents,
} from "leadtype/llm";
```

The CLI's default `leadtype generate` calls the website APIs; `leadtype generate --bundle` calls `generateAgentsMd`. Use these APIs directly when you need finer control.

## What gets generated

|File|Purpose|
|--|--|
|`<out>/llms.txt`|Top-level routing index. Product summary, best starting points, and root-relative markdown mirror links.|
|`<out>/docs/llms.txt`|Docs-scoped routing index. Lists every group with descriptions and markdown entry links.|
|`<out>/llms-full.txt`|All generated markdown docs flattened into one fallback file. Use when page-level links are not enough.|
|`<out>/docs/sitemap.xml`|Docs-scoped XML sitemap with canonical page URLs and `<lastmod>` dates.|
|`<out>/docs/sitemap.md`|Docs-scoped markdown sitemap with headings and links.|
|`<out>/docs/robots.txt`|Mergeable crawl policy that explicitly allows common AI crawlers.|
|`<out>/docs/agent-readability.json`|Structured manifest for host apps that need to merge docs pages with marketing, blog, changelog, or product pages.|

Group metadata still drives `llms.txt` sections, navigation, search metadata, and `AGENTS.md`, but site mode no longer emits per-group full-context files by default. See [Evals](/docs/reference/evals) for the benchmark behind that default.

## ProductInfo

Every generator that writes a top-level index file (`generateLlmsTxt`, `generateLLMFullContextFiles`, `generateAgentReadabilityArtifacts`, `generateAgentsMd`) takes the same `product` object:

|Property|Type|Description|Default|Required|
|:--|:--|:--|:--|:--:|
|name|string|Product display name. Rendered as the H1 of llms.txt and AGENTS.md.|-|✅ Required|
|summary|string|Single-line summary rendered as the blockquote under the H1.|-|✅ Required|
|bullets|string\[]|Rendered under '## Product Summary'. Use for top-level feature highlights.|-|Optional|
|bestStartingPoints|Array\<\{ urlPath: string, title?: string, description?: string }>|Curated entry points rendered under '## Best Starting Points'. Titles and descriptions fall back to each page's frontmatter when omitted.|-|Optional|
|agentGuidance|string|Optional routing instruction appended at the bottom of llms.txt. Ignored by generateAgentsMd — the URL-routing flow doesn't apply to offline package docs.|-|Optional|

The `docs.config.ts` `product` field has this exact shape; both the CLI and the library APIs read it.

## Example llms.txt

```txt
# My Library

> A library that does one thing well.

- Helper that handles the boring parts.
- Type-safe by default.
- Works in any runtime.

## Best Starting Points

- [Documentation](/docs/index.md)
- [Quickstart](/docs/quickstart.md)

## Get Started

Five-minute happy path and the mental model.

- [Quickstart](/docs/quickstart.md): Install and run the pipeline.
- [How it works](/docs/how-it-works.md): The mental model.

## Reference

CLI flags and conversion APIs.

- [CLI](/docs/reference/cli.md): Every flag.
```

## Typical sequence

```ts
import { convertAllMdx } from "leadtype/convert";
import { defaultRemarkPlugins, remarkInclude } from "leadtype/remark";
import {
  generateAgentReadabilityArtifacts,
  generateLLMFullContextFiles,
  generateLlmsTxt,
} from "leadtype/llm";

await convertAllMdx({
  srcDir: "docs",
  outDir: "public/docs",
  remarkPlugins: [remarkInclude, ...defaultRemarkPlugins],
});

await generateLlmsTxt({
  srcDir: ".",
  outDir: "public",
  baseUrl: "https://leadtype.dev",
  product: {
    name: "My Library",
    summary: "A library that does one thing well.",
  },
  groups: docsConfig.groups,
});

await generateLLMFullContextFiles({
  outDir: "public",
  baseUrl: "https://leadtype.dev",
  product: { name: "My Library" },
  groups: docsConfig.groups,
});

await generateAgentReadabilityArtifacts({
  outDir: "public",
  baseUrl: "https://leadtype.dev",
  product: docsConfig.product,
  groups: docsConfig.groups,
});
```

`generateLLMFullContextFiles` and `generateAgentReadabilityArtifacts` read from `<outDir>/docs/`, so always run conversion first.

## Mounted URL prefixes

By default, generated markdown under `<outDir>/docs/` maps to public URLs under `/docs`. Use `mounts` when part of that generated tree should have a different canonical URL, such as a changelog that lives beside docs in the source repo:

```ts
const mounts = [
  { pathPrefix: "", urlPrefix: "/docs" },
  { pathPrefix: "changelog", urlPrefix: "/changelog" },
];
```

With those mounts, `public/docs/quickstart.md` is still `/docs/quickstart.md`, while `public/docs/changelog/v1.md` is described as `/changelog/v1.md` in `llms.txt`, `llms-full.txt`, navigation, sitemap data, Agent Readability metadata, and search metadata.

Pass the same `mounts` array to every artifact generator that reads generated markdown:

```ts
await generateLlmsTxt({
  srcDir: ".",
  outDir: "public",
  product: docsConfig.product,
  groups: docsConfig.groups,
  mounts,
});

await generateLLMFullContextFiles({
  outDir: "public",
  product: { name: docsConfig.product.name },
  groups: docsConfig.groups,
  mounts,
});

await generateAgentReadabilityArtifacts({
  outDir: "public",
  product: docsConfig.product,
  groups: docsConfig.groups,
  mounts,
});
```

`resolveDocsNavigation` and `resolveDocsTableOfContents` also accept `mounts` so sidebars and page metadata use the same URL paths. If you use the CLI, prefer `--docs-dir changelog=/changelog`; it creates the equivalent mounts and also writes static markdown mirrors at `public/changelog/*.md`.

## generateAgentReadabilityArtifacts

For hosted docs sites that need the Vercel Agent Readability discovery layer without letting leadtype own the whole website:

```ts
import {
  generateAgentReadabilityArtifacts,
  renderRobotsTxt,
  renderSitemapMarkdown,
  renderSitemapXml,
} from "leadtype/llm";

const result = await generateAgentReadabilityArtifacts({
  outDir: "public",
  baseUrl: "https://leadtype.dev",
  product: docsConfig.product,
  groups: docsConfig.groups,
});
```

Writes docs-scoped files under `<outDir>/docs/`. The returned manifest contains the docs pages, markdown mirror URLs, group navigation, and freshness dates. Host apps can then merge `result.manifest.pages` with non-docs routes before serving root-level `/sitemap.xml`, `/sitemap.md`, and `/robots.txt`.

```ts
await writeFile("public/sitemap.xml", renderSitemapXml(result.manifest.pages));
await writeFile(
  "public/sitemap.md",
  renderSitemapMarkdown({
    product: docsConfig.product,
    navigation: result.manifest.navigation,
    pages: result.manifest.pages,
  })
);
await writeFile(
  "public/robots.txt",
  renderRobotsTxt({ baseUrl: "https://leadtype.dev" })
);
```

## Agent readability helpers

`generateAgentReadabilityArtifacts` gives you the manifest. The runtime helpers turn it into Web `Response` objects your framework can return directly:

```ts
import {
  acceptsMarkdownHeader,
  createAgentMarkdownResponse,
  createDocsHead,
  createMarkdownResponseHeaders,
  createRobotsTxtResponse,
  createSitemapMarkdownResponse,
  createSitemapXmlResponse,
  enrichMarkdownFrontmatter,
  isAgentReadabilityArtifactPath,
  isAgentUserAgent,
  renderJsonLd,
  renderJsonLdScript,
  renderMissingMarkdown,
  resolveMarkdownMirrorTarget,
} from "leadtype/llm/readability";
```

The whole module is fs-free and edge-runtime safe: it works in Node, Bun, Vercel Edge, Cloudflare Workers, and any framework with a Web `Request`/`Response` API.

### createAgentMarkdownResponse

Markdown content negotiation for docs pages. Returns `Response | null` (`null` means the path is not an agent-oriented markdown request — fall through to your HTML route). `readMarkdownFile` may be sync or async.

```ts
const response = await createAgentMarkdownResponse({
  urlPath: request.url,
  method: request.method,
  headers: Object.fromEntries(request.headers),
  manifest,
  requestOrigin: new URL(request.url).origin,
  async readMarkdownFile(target) {
    return await readGeneratedFile(`public/${target.filePath}`);
  },
});

if (response) {
  return response;
}
```

Headers it sets:

* `Content-Type: text/markdown; charset=utf-8`
* `Vary: Accept` (and `, User-Agent` when an AI UA is detected)
* `Link: <canonical>; rel="canonical"`
* `Cache-Control: public, max-age=300, must-revalidate` (override with `cacheControl: "<directive>"` or omit with `cacheControl: null`)

It also enriches frontmatter with `canonical_url` and `last_updated`, leaves discovery artifacts (`llms.txt`, `sitemap.*`, `robots.txt`, `agent-readability.json`, root `llms-full.txt`, search JSON) alone, and returns a 200 markdown "Page not found" body only for agent-oriented requests.

Optional `userAgentPattern` overrides the default AI user-agent regex (defaults cover GPTBot, ChatGPT-User, OAI-SearchBot, Claude-Web, ClaudeBot, anthropic-ai, CCBot, Google-Extended, AmazonBot, Bingbot, MetaExternalAgent, ByteSpider, PerplexityBot, MistralBot, AppleBot, YouBot).

### createSitemapXmlResponse / createSitemapMarkdownResponse / createRobotsTxtResponse

Build sitemap and robots responses at request time so the `<loc>` and `Sitemap:` directives reflect the live origin (preview, staging, prod), no string-replace hacks needed.

```ts
const sitemap = createSitemapXmlResponse({
  manifest,
  requestOrigin: new URL(request.url).origin,
  // Optional: merge non-docs pages into the sitemap before rebasing.
  pages: [...manifest.pages, ...marketingPages],
});

const robots = createRobotsTxtResponse({
  manifest,
  requestOrigin: new URL(request.url).origin,
  // Optional: docs-scoped robots points to /docs/sitemap.xml.
  sitemapUrlPath: "/docs/sitemap.xml",
});
```

`requestOrigin` is optional — when omitted, the helpers fall back to `manifest.baseUrl`.

### createDocsHead

Build canonical/alternate/og/JSON-LD head metadata for a docs page from the manifest. Returns `{ meta, links }` in a framework-neutral shape (TanStack Router, Next.js Metadata, Astro, etc.).

```ts
import manifest from "../public/docs/agent-readability.json";
import { createDocsHead } from "leadtype/llm/readability";

const head = createDocsHead({
  urlPath: "/docs/quickstart",
  manifest,
  // Optional: override the JSON-LD meta key (default: "script:ld+json"
  // for TanStack Router; use "ldJson" for typed Metadata APIs).
  jsonLdMetaKey: "script:ld+json",
});

// head.meta = [
//   { title: "Quickstart | Leadtype" },
//   { name: "description", content: "Install and run." },
//   { property: "og:title", content: "Quickstart | Leadtype" },
//   { property: "og:description", content: "Install and run." },
//   { "script:ld+json": { "@context": "https://schema.org", ... } },
// ];
// head.links = [
//   { rel: "canonical", href: "https://leadtype.dev/docs/quickstart" },
//   { rel: "alternate", type: "text/markdown",
//     href: "https://leadtype.dev/docs/quickstart.md" },
// ];
```

If the page is not in the manifest, both arrays are empty so the caller can fall back to its own metadata.

### Lower-level helpers

For a step-by-step integration guide, see [Optimize docs for agents](/docs/build/optimize-docs-for-agents).

|Helper|Use it for|
|--|--|
|`renderJsonLd`|Return a Schema.org object for framework metadata APIs.|
|`renderJsonLdScript`|Return an escaped `<script type="application/ld+json">` string.|
|`createAgentMarkdownResponse`|Build a complete markdown `Response` for agent Accept headers, AI user agents, and direct `.md` URLs.|
|`createSitemapXmlResponse` / `createSitemapMarkdownResponse` / `createRobotsTxtResponse`|Build sitemap/robots `Response`s rebased to the live origin.|
|`createDocsHead`|Build canonical/alternate/og/JSON-LD head entries from the manifest.|
|`resolveMarkdownMirrorTarget`|Map `/docs/foo` or `/docs/foo.md` to the generated `docs/foo.md` file path.|
|`createMarkdownResponseHeaders`|Produce canonical markdown headers with `Content-Type`, `Vary`, `Link`, `Cache-Control`.|
|`enrichMarkdownFrontmatter`|Add `canonical_url` and `last_updated` aliases to generated markdown (CRLF-tolerant).|
|`renderMissingMarkdown`|Render the 200 markdown body used for missing docs pages requested by agents.|
|`acceptsMarkdownHeader`|Detect whether an `Accept` header is asking for markdown (q-value aware).|
|`isAgentUserAgent`|Detect known AI crawler and agent user agents. Accepts an optional regex override.|
|`isAgentReadabilityArtifactPath`|Avoid rewriting `llms.txt`, `llms-full.txt`, sitemap, robots, search JSON, and manifest artifact paths.|

### Cache-Control and CDN

Every helper sets `Cache-Control: public, max-age=300, must-revalidate` by default and pairs it with `Vary: Accept` (and `, User-Agent` for AI UAs). When you cache responses on a CDN, make sure it shards by `Accept` and `User-Agent` so agents and browsers don't cross-pollinate cached HTML and markdown variants. Pass `cacheControl: "<directive>"` to override or `cacheControl: null` to omit the header (when you set caching at the CDN layer).

### Manifest version

Every helper asserts `manifest.version === 1` at the entry point and throws a clear error otherwise — so a stale dev `agent-readability.json` from a different leadtype version fails loudly instead of silently producing wrong responses.

## generateAgentsMd

For npm-bundled docs that ship inside `node_modules/<pkg>/` and need relative filesystem links:

```ts
import { generateAgentsMd } from "leadtype/llm";

await generateAgentsMd({
  srcDir: ".",
  outDir: "packages/my-package",
  product: docsConfig.product,
  groups: docsConfig.groups,
});
```

Writes `<outDir>/AGENTS.md` (singular, at the root) with the same product summary and group structure as `generateLlmsTxt`, but every link is a **relative** path like `./docs/<segment>/<slug>.md` instead of an absolute URL.

|Option|Description|
|--|--|
|`srcDir`|Repo root containing the `docs/` source.|
|`outDir`|Package root. `AGENTS.md` is written at `<outDir>/AGENTS.md`.|
|`product`|Same `ProductInfo` shape as `generateLlmsTxt`. `agentGuidance` is intentionally ignored — it's written for the website's URL-routing flow and would mislead an offline agent.|
|`groups`|Group tree from `docs.config.ts`. Same shape as `generateLlmsTxt`.|
|`docsSubdir`|Subdirectory under `outDir` that holds the `.md` files. Default: `docs`. Used as the relative-path prefix in every link.|

Returns `{ outputPath: string }` pointing at the written `AGENTS.md`.

### Example output

```md
# My Library

> A library that does one thing well.

These docs ship inside the package so coding agents can read them offline. Open the topic file you need from the list below — paths are relative to this file.

## Get Started

- [Quickstart](./docs/quickstart.md): Install and run the pipeline.
- [How it works](./docs/how-it-works.md): The mental model.

## Reference

- [CLI](./docs/reference/cli.md): Every flag.
```

Pair with `convertAllMdx` to produce the `.md` files the links resolve to. See [Bundle docs into a package](/docs/package-docs/bundle) for the full flow.

## resolveDocsNavigation

Same group-resolution logic the LLM files use, but returns the navigation manifest as a plain object — useful for driving a sidebar UI:

```ts
const navigation = await resolveDocsNavigation({
  srcDir: ".",
  baseUrl: "https://leadtype.dev",
  groups: docsConfig.groups,
});

if (navigation.unknown.length > 0) {
  for (const { urlPath, slug } of navigation.unknown) {
    process.stderr.write(`error: ${urlPath} declares unknown group "${slug}".\n`);
  }
  process.exit(1);
}
```

Write the result to `src/generated/docs-nav.json` and import it from your sidebar component:

```ts
import { mkdir, writeFile } from "node:fs/promises";

await mkdir("src/generated", { recursive: true });
await writeFile(
  "src/generated/docs-nav.json",
  `${JSON.stringify(navigation, null, 2)}\n`
);
```

Now your sidebar can import a static manifest with the same group tree the LLM files use.

## Tables of contents

For the heading slug contract and renderer wiring, see [Render MDX and TOC](/docs/build/render-mdx-and-toc). This section covers the build-time APIs only.

`resolveDocsNavigation` includes a `toc` array on every page by default. The default range is `h2`–`h3`, which keeps page-level `h1` titles out of sidebars.

```ts
const navigation = await resolveDocsNavigation({
  srcDir: ".",
  baseUrl: "https://leadtype.dev",
  groups: docsConfig.groups,
  toc: { minLevel: 2, maxLevel: 4 },
});
```

If you only need TOC data and not the full group tree, call `resolveDocsTableOfContents`:

```ts
const pages = await resolveDocsTableOfContents({
  srcDir: ".",
  baseUrl: "https://leadtype.dev",
});
```

For custom pipelines, `extractDocsTableOfContents` accepts a markdown or MDX string plus page URL metadata and returns plain JSON. It ignores frontmatter and fenced code blocks, and it uses the same `slugifyDocsHeading` helper that rendered headings must use to keep `id` attributes in sync.

## Group design

The `groups` you pass to these APIs come from `docs.config.ts`. Two principles:

* **Use groups for routing, not sharding.** Groups organize `llms.txt`, navigation, search metadata, and `AGENTS.md`. The root `llms-full.txt` remains the broad fallback.
* **Write group descriptions for routing, not flavor text.** Agents read those descriptions to decide which pages to load. "How to install and run" beats "Welcome to our guides!"

## Base URL precedence

Pass `baseUrl` explicitly, or use environment variables for layered fallback:

```ts
const baseUrl =
  process.env.LEADTYPE_AGENT_BASE_URL ||
  process.env.BASE_URL ||
  process.env.PORTLESS_URL ||
  "https://leadtype.dev";
```

The package-specific `LEADTYPE_AGENT_BASE_URL` lets each package override an org-wide default. `BASE_URL` covers most CI/deployment platforms, and a final hardcoded fallback keeps local builds working without env setup.