Evals

Leadtype treats agent-facing docs as behavior, not just files on disk. The repo ships eval harnesses (under evals/) that run real coding agents against generated artifacts and grade them with an independent LLM judge.

Read this if you want the evidence behind three claims: that bundling docs into a package measurably helps coding agents, that the default llms.txt + llms-full.txt shape routes agents most reliably, and that agents rarely discover llms.txt on their own unless pointed at it.

How the harness works

Each eval is one cell — a fixture (task) × arm × model — run 10 times. A run:

1. Spins up an isolated temp project and npm installs a packed leadtype tarball, so node_modules/leadtype/ looks like a real install.
2. Lets the agent work with a small set of path-scoped tools (read, write, list, glob, grep, and a narrow npm). No shell, no network beyond the model call.
3. Sends the agent's answer (and any files it wrote) to an LLM judge, which grades it against a per-fixture rubric and returns pass/fail + a 0–100 score + a failure-mode classification. We judge with deepseek-v4-pro — a model from a family with no candidate under test, so it has no same-family preference. The judge is the headline metric; "did the agent read our files" is tracked separately as supporting evidence.

Pass rates are reported with a Wilson 95% confidence interval, and every run's transcript, judge verdict, and produced files are archived under evals/results/ for inspection. Except where noted (the cost table was re-measured 2026-06-01 after a token-counting fix), the numbers below are from the 2026-05-31 run: the package benchmark uses 5 models across 4 families (Anthropic Haiku 4.5 / Opus 4.8, OpenAI GPT‑5.5, Moonshot Kimi K2.6, Google Gemini 3.5 Flash) × 4 arms × 10 runs, judged by deepseek-v4-pro and cross-validated with grok-4.3. The hosted-docs benchmarks use a 3-model subset (Haiku, Opus, GPT‑5.5).

Does bundling docs help? (package benchmark)

The package benchmark answers the core question with an A/B test:

• Treatment — the installed leadtype ships its bundled AGENTS.md + docs/*.md.
• Control — those files are stripped (along with dist/*.map source maps, which would otherwise leak the original commented source). The compiled JS — including the CLI's --help text — and the .d.ts types stay. An honest "a package that simply didn't bundle agent docs," not "an agent with no information."

Three things change when the docs are present, and they differ in how universal they are. Two hold for every model and are the wins to lead with; the third is real but uneven:

1. Every run gets cheaper — universal, frontier models included.
2. Agents stop confidently guessing wrong about the API — the clearest, most judge-robust correctness win.
3. Raw accuracy rises — but mostly for the small, cheap models most agents run on; the frontier lift is small and judge-sensitive.

Every run gets cheaper

Pooled across fixtures, bundling docs cut the tokens every model spent — the agent reads one short doc instead of probing the package with repeated grep/read/list calls. Tokens (input + output, summed across every step of the tool loop) drop 32–54%, and tool calls drop for every model too:

Even GPT‑5.5, which barely needs docs for correctness, spent 32% fewer tokens with them — the token win holds regardless of model tier or whether docs ever move the pass rate. Wall-clock is not a claim we lean on: it tracks gateway latency more than work done, and is unstable run-to-run (Haiku came out slower with docs here, −37% in an earlier run). Tokens and tool calls are the defensible cost signal.

Agents stop confidently guessing wrong

The judge classifies each failure, and the dangerous mode is confidently wrong — asserting the opposite of the truth with no hedging. Docs cut it across the board:

The clearest case is nav-unknown-group — "what happens if a page declares a group the config doesn't know?" The build actually fails, but without docs agents don't say "I don't know"; they confidently assert the wrong answer. Bundling docs is cheap insurance against exactly that, and it's the most defensible reason to ship them — it doesn't depend on the model being small or the judge being generous.

Accuracy lift — concentrated in small models

Bundling docs also lifts raw task success, but the gain is largest for the smaller, cheaper models that most agents actually run on, and marginal for frontier models that recover more from the package alone:

(Pooled over all fixtures, n = 60 runs per cell, judged by neutral deepseek-v4-pro.)

Decomposing the value: code vs. docs vs. memory

A bare arm installs nothing (pure training-memory recall); control adds the compiled package; treatment adds the docs; pointer adds leadtype's recommended root AGENTS.md pointer. The ladder shows exactly what each layer buys:

Small models are nearly helpless from memory (Haiku 22%) and gain from both the installed code and the docs; frontier models already know ~90% and docs add the last few points.

Where the lift comes from

The per-fixture breakdown is more telling than the average: almost all of the accuracy lift comes from two behavioral gotchas the compiled package can't self-document.

• nav-unknown-group (+28 pooled) — the confident-wrong case above; docs turn a confident wrong answer into a correct one.
• search-when-embeddings (+28 pooled) — "when is the default static BM25 index enough vs. embeddings?" Another non-obvious behavioral rule recoverable only from prose.
• explain-cli-flag, mounted-changelog-urls, bundle-rationale, custom-generate-script — roughly flat (−2 to +6). Recoverable from the CLI's --help, the .d.ts types, or the README. We keep them to show the benchmark isn't cherry-picked and to pin down where docs don't pay off.

So docs add little for facts your CLI, types, or README already expose — the accuracy payoff is concentrated where the library's behavior is non-obvious, and largest for smaller models. Write for agents turns this into an authoring rule.

Judge choice matters

LLM judges can favor their own family. Once Gemini was a candidate, the judge had to come from a family with no candidate, so we grade with deepseek-v4-pro and re-graded the headline arms with a second neutral judge, grok-4.3:

Both neutral judges agree on the direction and rank order — small models gain most, frontier least. Frontier magnitudes are judge-sensitive (Opus +7 vs +0) but small either way, so the qualitative claim holds. Reproduce with bun run rejudge <runDir> --judge xai/grok-4.3 --arms treatment,control.

Which llms.txt shape routes agents best? (hosted-docs benchmark)

The hosted-docs benchmark simulates a hosted docs web root as local files (3-model subset). Agents start at /llms.txt and follow links; a "pass" means the judge marked the answer correct, and context match means the agent actually read the context path the variant intends. A shape only earns trust when both are high.

What we found

Pass rate is similar across shapes, so the discriminating metric is context match — did the agent follow the path the shape intends?

page-links, monolith, and section-indexes are the shapes agents reliably use as intended. explicit-bundles and router look fine on pass rate but agents don't follow their intended path — they answer from the /llms.txt summary or grab whatever bundle looks closest. The extra structure doesn't earn its keep.

Agents rarely discover llms.txt on their own

The routing benchmark tells agents to start at /llms.txt. A separate discovery arm drops that hint and serves a realistic web root (docs pages + llms.txt + llms-full.txt + sitemap + robots). Agents then consult /llms.txt only ~29% of the time — they mostly grep the doc pages directly and still answer correctly. So llms.txt earns its keep most as a pointed-to entry (a root AGENTS.md, or a tool that fetches it), not as an organically-discovered one.

Current defaults

Leadtype keeps the public website artifact set small:

public/
├── llms.txt
├── llms-full.txt
└── docs/*.md

/llms.txt routes agents to page-level markdown first; /llms-full.txt is the broad all-docs fallback. Groups organize navigation, llms.txt sections, search metadata, and AGENTS.md; they aren't published as per-group full-context files by default.

For packages, leadtype generate --bundle ships AGENTS.md + docs/*.md inside the tarball — and the recommended consumer setup is a root AGENTS.md pointing at node_modules/leadtype/AGENTS.md, which the benchmark's pointer arm shows beats leaving the agent to find the bundle on its own.

Run the evals

cd evals
bun install && bun run pack-leadtype
bun run evals:full:arms       # package: 5 models × bare/control/treatment/pointer × 10
bun run evals:llms:full        # hosted-docs routing (3-model subset)
bun run evals:llms:discovery   # unhinted llms.txt discovery

The detailed harness docs live in evals/README.md.