LLM files

The leadtype/llm entry point produces four flavors of agent-facing output, all derived from the same docs source:

generateLlmsTxt — for hosted websites. Emits the /llms.txt convention with root-relative markdown mirror links.
generateLLMFullContextFiles — root /llms-full.txt fallback containing all generated markdown docs. Pairs with generateLlmsTxt.
generateAgentReadabilityArtifacts — docs-scoped sitemap.xml, sitemap.md, robots.txt, and JSON manifest data that a host app can merge into site-level files.
generateAgentsMd — for npm-bundled docs. Emits an AGENTS.md index with relative ./docs/<topic>.md paths so the file works inside node_modules/<pkg>/.

import {
  extractDocsTableOfContents,
  generateAgentReadabilityArtifacts,
  generateAgentsMd,
  generateLLMFullContextFiles,
  generateLlmsTxt,
  renderRobotsTxt,
  renderSitemapMarkdown,
  renderSitemapXml,
  resolveDocsNavigation,
  resolveDocsTableOfContents,
} from 'leadtype/llm';

The CLI's default leadtype generate calls the website APIs; leadtype generate --bundle calls generateAgentsMd. Use these APIs directly when you need finer control.

What gets generated

File	Purpose
`<out>/llms.txt`	Top-level routing index. Product summary, best starting points, and root-relative markdown mirror links.
`<out>/docs/llms.txt`	Docs-scoped routing index. Lists every group with descriptions and markdown entry links.
`<out>/llms-full.txt`	All generated markdown docs flattened into one fallback file. Use when page-level links are not enough.
`<out>/docs/sitemap.xml`	Docs-scoped XML sitemap with canonical page URLs and `<lastmod>` dates.
`<out>/docs/sitemap.md`	Docs-scoped markdown sitemap with headings and links.
`<out>/docs/robots.txt`	Mergeable crawl policy that explicitly allows common AI crawlers.
`<out>/docs/agent-readability.json`	Structured manifest for host apps that need to merge docs pages with marketing, blog, changelog, or product pages.

Group metadata still drives llms.txt sections, navigation, search metadata, and AGENTS.md, but site mode no longer emits per-group full-context files by default. See Evals for the benchmark behind that default.

ProductInfo

Every generator that writes a top-level index file (generateLlmsTxt, generateLLMFullContextFiles, generateAgentReadabilityArtifacts, generateAgentsMd) takes the same product object:

Name	Type	Default	Description
`name`required	`string`	-	Product display name. Rendered as the H1 of llms.txt and AGENTS.md.
`summary`required	`string`	-	Single-line summary rendered as the blockquote under the H1.
`bullets`	`string[]`	-	Rendered under '## Product Summary'. Use for top-level feature highlights.
`bestStartingPoints`	`Array<{ urlPath: string, title?: string, description?: string }>`	-	Curated entry points rendered under '## Best Starting Points'. Titles and descriptions fall back to each page's frontmatter when omitted.
`agentGuidance`	`string`	-	Optional routing instruction appended at the bottom of llms.txt. Ignored by generateAgentsMd — the URL-routing flow doesn't apply to offline package docs.

The docs.config.ts product field has this exact shape; both the CLI and the library APIs read it.

Example llms.txt

# My Library

> A library that does one thing well.

- Helper that handles the boring parts.
- Type-safe by default.
- Works in any runtime.

## Best Starting Points

- [Documentation](/docs/index.md)
- [Quickstart](/docs/quickstart.md)

## Get Started

Five-minute happy path and the mental model.

- [Quickstart](/docs/quickstart.md): Install and run the pipeline.
- [How it works](/docs/how-it-works.md): The mental model.

## Reference

CLI flags and conversion APIs.

- [CLI](/docs/reference/cli.md): Every flag.

Typical sequence

import { convertAllMdx } from 'leadtype/convert';
import { defaultRemarkPlugins, remarkInclude } from 'leadtype/remark';
import {
  generateAgentReadabilityArtifacts,
  generateLLMFullContextFiles,
  generateLlmsTxt,
} from 'leadtype/llm';

await convertAllMdx({
  srcDir: 'docs',
  outDir: 'public/docs',
  remarkPlugins: [remarkInclude, ...defaultRemarkPlugins],
});

await generateLlmsTxt({
  srcDir: '.',
  outDir: 'public',
  baseUrl: 'https://leadtype.dev',
  product: {
    name: 'My Library',
    summary: 'A library that does one thing well.',
  },
  groups: docsConfig.groups,
});

await generateLLMFullContextFiles({
  outDir: 'public',
  baseUrl: 'https://leadtype.dev',
  product: { name: 'My Library' },
  groups: docsConfig.groups,
});

await generateAgentReadabilityArtifacts({
  outDir: 'public',
  baseUrl: 'https://leadtype.dev',
  product: docsConfig.product,
  groups: docsConfig.groups,
});

generateLLMFullContextFiles and generateAgentReadabilityArtifacts read from <outDir>/docs/, so always run conversion first.

Mounted URL prefixes

By default, generated markdown under <outDir>/docs/ maps to public URLs under /docs. Use mounts when part of that generated tree should have a different canonical URL, such as a changelog that lives beside docs in the source repo:

const mounts = [
  { pathPrefix: '', urlPrefix: '/docs' },
  { pathPrefix: 'changelog', urlPrefix: '/changelog' },
];

With those mounts, public/docs/quickstart.md is still /docs/quickstart.md, while public/docs/changelog/v1.md is described as /changelog/v1.md in llms.txt, llms-full.txt, navigation, sitemap data, Agent Readability metadata, and search metadata.

Pass the same mounts array to every artifact generator that reads generated markdown:

await generateLlmsTxt({
  srcDir: '.',
  outDir: 'public',
  product: docsConfig.product,
  groups: docsConfig.groups,
  mounts,
});

await generateLLMFullContextFiles({
  outDir: 'public',
  product: { name: docsConfig.product.name },
  groups: docsConfig.groups,
  mounts,
});

await generateAgentReadabilityArtifacts({
  outDir: 'public',
  product: docsConfig.product,
  groups: docsConfig.groups,
  mounts,
});

resolveDocsNavigation and resolveDocsTableOfContents also accept mounts so sidebars and page metadata use the same URL paths. If you use the CLI, prefer --docs-dir changelog=/changelog; it creates the equivalent mounts and also writes static markdown mirrors at public/changelog/*.md.

generateAgentReadabilityArtifacts

For hosted docs sites that need the Vercel Agent Readability discovery layer without letting leadtype own the whole website:

import {
  generateAgentReadabilityArtifacts,
  renderRobotsTxt,
  renderSitemapMarkdown,
  renderSitemapXml,
} from 'leadtype/llm';

const result = await generateAgentReadabilityArtifacts({
  outDir: 'public',
  baseUrl: 'https://leadtype.dev',
  product: docsConfig.product,
  groups: docsConfig.groups,
});

Writes docs-scoped files under <outDir>/docs/. The returned manifest contains the docs pages, markdown mirror URLs, group navigation, and freshness dates. Host apps can then merge result.manifest.pages with non-docs routes before serving root-level /sitemap.xml, /sitemap.md, and /robots.txt.

await writeFile('public/sitemap.xml', renderSitemapXml(result.manifest.pages));
await writeFile(
  'public/sitemap.md',
  renderSitemapMarkdown({
    product: docsConfig.product,
    navigation: result.manifest.navigation,
    pages: result.manifest.pages,
  })
);
await writeFile(
  'public/robots.txt',
  renderRobotsTxt({ baseUrl: 'https://leadtype.dev' })
);

Agent readability helpers

generateAgentReadabilityArtifacts gives you the manifest. The runtime helpers turn it into Web Response objects your framework can return directly:

import {
  acceptsMarkdownHeader,
  createAgentMarkdownResponse,
  createDocsHead,
  createMarkdownResponseHeaders,
  createRobotsTxtResponse,
  createSitemapMarkdownResponse,
  createSitemapXmlResponse,
  enrichMarkdownFrontmatter,
  isAgentReadabilityArtifactPath,
  isAgentUserAgent,
  renderJsonLd,
  renderJsonLdScript,
  renderMissingMarkdown,
  resolveMarkdownMirrorTarget,
} from 'leadtype/llm/readability';

The whole module is fs-free and edge-runtime safe: it works in Node, Bun, Vercel Edge, Cloudflare Workers, and any framework with a Web Request/Response API.

createAgentMarkdownResponse

Markdown content negotiation for docs pages. Returns Response | null (null means the path is not an agent-oriented markdown request — fall through to your HTML route). readMarkdownFile may be sync or async.

const response = await createAgentMarkdownResponse({
  urlPath: request.url,
  method: request.method,
  headers: Object.fromEntries(request.headers),
  manifest,
  requestOrigin: new URL(request.url).origin,
  async readMarkdownFile(target) {
    return await readGeneratedFile(`public/${target.filePath}`);
  },
});

if (response) {
  return response;
}

Headers it sets:

Content-Type: text/markdown; charset=utf-8
Vary: Accept (and , User-Agent when an AI UA is detected)
Link: <canonical>; rel="canonical"
Cache-Control: public, max-age=300, must-revalidate (override with cacheControl: "<directive>" or omit with cacheControl: null)

It also enriches frontmatter with canonical_url and last_updated, leaves discovery artifacts (llms.txt, sitemap.*, robots.txt, agent-readability.json, root llms-full.txt, search JSON) alone, and returns a 200 markdown "Page not found" body only for agent-oriented requests.

Optional userAgentPattern overrides the default AI user-agent regex (defaults cover GPTBot, ChatGPT-User, OAI-SearchBot, Claude-Web, ClaudeBot, anthropic-ai, CCBot, Google-Extended, AmazonBot, Bingbot, MetaExternalAgent, ByteSpider, PerplexityBot, MistralBot, AppleBot, YouBot).

createSitemapXmlResponse / createSitemapMarkdownResponse / createRobotsTxtResponse

Build sitemap and robots responses at request time so the <loc> and Sitemap: directives reflect the live origin (preview, staging, prod), no string-replace hacks needed.

const sitemap = createSitemapXmlResponse({
  manifest,
  requestOrigin: new URL(request.url).origin,
  // Optional: merge non-docs pages into the sitemap before rebasing.
  pages: [...manifest.pages, ...marketingPages],
});

const robots = createRobotsTxtResponse({
  manifest,
  requestOrigin: new URL(request.url).origin,
  // Optional: docs-scoped robots points to /docs/sitemap.xml.
  sitemapUrlPath: '/docs/sitemap.xml',
});

requestOrigin is optional — when omitted, the helpers fall back to manifest.baseUrl.

createDocsHead

Build canonical/alternate/og/JSON-LD head metadata for a docs page from the manifest. Returns { meta, links } in a framework-neutral shape (TanStack Router, Next.js Metadata, Astro, etc.).

import manifest from '../public/docs/agent-readability.json';
import { createDocsHead } from 'leadtype/llm/readability';

const head = createDocsHead({
  urlPath: '/docs/quickstart',
  manifest,
  // Optional: override the JSON-LD meta key (default: "script:ld+json"
  // for TanStack Router; use "ldJson" for typed Metadata APIs).
  jsonLdMetaKey: 'script:ld+json',
});

// head.meta = [
//   { title: "Quickstart | Leadtype" },
//   { name: "description", content: "Install and run." },
//   { property: "og:title", content: "Quickstart | Leadtype" },
//   { property: "og:description", content: "Install and run." },
//   { "script:ld+json": { "@context": "https://schema.org", ... } },
// ];
// head.links = [
//   { rel: "canonical", href: "https://leadtype.dev/docs/quickstart" },
//   { rel: "alternate", type: "text/markdown",
//     href: "https://leadtype.dev/docs/quickstart.md" },
// ];

If the page is not in the manifest, both arrays are empty so the caller can fall back to its own metadata.

Lower-level helpers

For a step-by-step integration guide, see Optimize docs for agents.

Helper	Use it for
`renderJsonLd`	Return a Schema.org object for framework metadata APIs.
`renderJsonLdScript`	Return an escaped `<script type="application/ld+json">` string.
`createAgentMarkdownResponse`	Build a complete markdown `Response` for agent Accept headers, AI user agents, and direct `.md` URLs.
`createSitemapXmlResponse` / `createSitemapMarkdownResponse` / `createRobotsTxtResponse`	Build sitemap/robots `Response`s rebased to the live origin.
`createDocsHead`	Build canonical/alternate/og/JSON-LD head entries from the manifest.
`resolveMarkdownMirrorTarget`	Map `/docs/foo` or `/docs/foo.md` to the generated `docs/foo.md` file path.
`createMarkdownResponseHeaders`	Produce canonical markdown headers with `Content-Type`, `Vary`, `Link`, `Cache-Control`.
`enrichMarkdownFrontmatter`	Add `canonical_url` and `last_updated` aliases to generated markdown (CRLF-tolerant).
`renderMissingMarkdown`	Render the 200 markdown body used for missing docs pages requested by agents.
`acceptsMarkdownHeader`	Detect whether an `Accept` header is asking for markdown (q-value aware).
`isAgentUserAgent`	Detect known AI crawler and agent user agents. Accepts an optional regex override.
`isAgentReadabilityArtifactPath`	Avoid rewriting `llms.txt`, `llms-full.txt`, sitemap, robots, search JSON, and manifest artifact paths.

Cache-Control and CDN

Every helper sets Cache-Control: public, max-age=300, must-revalidate by default and pairs it with Vary: Accept (and , User-Agent for AI UAs). When you cache responses on a CDN, make sure it shards by Accept and User-Agent so agents and browsers don't cross-pollinate cached HTML and markdown variants. Pass cacheControl: "<directive>" to override or cacheControl: null to omit the header (when you set caching at the CDN layer).

Manifest version

Every helper asserts manifest.version === 1 at the entry point and throws a clear error otherwise — so a stale dev agent-readability.json from a different leadtype version fails loudly instead of silently producing wrong responses.

generateAgentsMd

For npm-bundled docs that ship inside node_modules/<pkg>/ and need relative filesystem links:

import { generateAgentsMd } from 'leadtype/llm';

await generateAgentsMd({
  srcDir: '.',
  outDir: 'packages/my-package',
  product: docsConfig.product,
  groups: docsConfig.groups,
});

Writes <outDir>/AGENTS.md (singular, at the root) with the same product summary and group structure as generateLlmsTxt, but every link is a relative path like ./docs/<segment>/<slug>.md instead of an absolute URL.

Option	Description
`srcDir`	Repo root containing the `docs/` source.
`outDir`	Package root. `AGENTS.md` is written at `<outDir>/AGENTS.md`.
`product`	Same `ProductInfo` shape as `generateLlmsTxt`. `agentGuidance` is intentionally ignored — it's written for the website's URL-routing flow and would mislead an offline agent.
`groups`	Group tree from `docs.config.ts`. Same shape as `generateLlmsTxt`.
`docsSubdir`	Subdirectory under `outDir` that holds the `.md` files. Default: `docs`. Used as the relative-path prefix in every link.

Returns { outputPath: string } pointing at the written AGENTS.md.

Example output

# My Library

> A library that does one thing well.

These docs ship inside the package so coding agents can read them offline. Open the topic file you need from the list below — paths are relative to this file.

## Get Started

- [Quickstart](./docs/quickstart.md): Install and run the pipeline.
- [How it works](./docs/how-it-works.md): The mental model.

## Reference

- [CLI](./docs/reference/cli.md): Every flag.

Pair with convertAllMdx to produce the .md files the links resolve to. See Bundle docs into a package for the full flow.

resolveDocsNavigation

Same group-resolution logic the LLM files use, but returns the navigation manifest as a plain object — useful for driving a sidebar UI:

const navigation = await resolveDocsNavigation({
  srcDir: '.',
  baseUrl: 'https://leadtype.dev',
  groups: docsConfig.groups,
});

if (navigation.unknown.length > 0) {
  for (const { urlPath, slug } of navigation.unknown) {
    process.stderr.write(
      `error: ${urlPath} declares unknown group "${slug}".\n`
    );
  }
  process.exit(1);
}

Write the result to src/generated/docs-nav.json and import it from your sidebar component:

import { mkdir, writeFile } from 'node:fs/promises';

await mkdir('src/generated', { recursive: true });
await writeFile(
  'src/generated/docs-nav.json',
  `${JSON.stringify(navigation, null, 2)}\n`
);

Now your sidebar can import a static manifest with the same group tree the LLM files use.

Tables of contents

For the heading slug contract and renderer wiring, see Render MDX and TOC. This section covers the build-time APIs only.

resolveDocsNavigation includes a toc array on every page by default. The default range is h2–h3, which keeps page-level h1 titles out of sidebars.

const navigation = await resolveDocsNavigation({
  srcDir: '.',
  baseUrl: 'https://leadtype.dev',
  groups: docsConfig.groups,
  toc: { minLevel: 2, maxLevel: 4 },
});

If you only need TOC data and not the full group tree, call resolveDocsTableOfContents:

const pages = await resolveDocsTableOfContents({
  srcDir: '.',
  baseUrl: 'https://leadtype.dev',
});

For custom pipelines, extractDocsTableOfContents accepts a markdown or MDX string plus page URL metadata and returns plain JSON. It ignores frontmatter and fenced code blocks, and it uses the same slugifyDocsHeading helper that rendered headings must use to keep id attributes in sync.

Group design

The groups you pass to these APIs come from docs.config.ts. Two principles:

Use groups for routing, not sharding. Groups organize llms.txt, navigation, search metadata, and AGENTS.md. The root llms-full.txt remains the broad fallback.
Write group descriptions for routing, not flavor text. Agents read those descriptions to decide which pages to load. "How to install and run" beats "Welcome to our guides!"

Base URL precedence

Pass baseUrl explicitly, or use environment variables for layered fallback:

const baseUrl =
  process.env.LEADTYPE_AGENT_BASE_URL ||
  process.env.BASE_URL ||
  process.env.PORTLESS_URL ||
  'https://leadtype.dev';

The package-specific LEADTYPE_AGENT_BASE_URL lets each package override an org-wide default. BASE_URL covers most CI/deployment platforms, and a final hardcoded fallback keeps local builds working without env setup.