AGENTS.md vs Skills for Teaching Agents Framework Docs

Vercel published eval results this week showing how framework authors should give coding agents access to their APIs. The team built an eval suite around Next.js 16 tasks and compared two ways of giving agents access to version-matched documentation: a skill the agent could invoke on demand, and a compressed docs index embedded directly in AGENTS.md. The results were lopsided. The AGENTS.md index hit a 100% pass rate. Skills topped out at 79%, only after adding explicit instructions telling the agent to use them. Without those instructions, the skill performed no better than shipping no documentation at all. Jude Gao conducted the research and evals.

Teams building with coding agents must decide whether to package needed knowledge as a tool the agent chooses to use or pin it into context so no choice is required.

Why the agents needed docs in the first place

Model training data goes stale, and framework APIs move faster than model releases. Next.js 16 introduced APIs like the 'use cache' directive, connection(), and forbidden() that aren't in current model training data. An agent that doesn't know these APIs generates incorrect code or falls back to older patterns it does know. On a project pinned to an older Next.js version, the model suggests newer APIs that don't exist there yet.

Vercel's goal was version-matched documentation, so the agent works from docs that match the project in front of it rather than from whatever snapshot of the framework existed at training time.

Before trusting any comparison, the team hardened the eval suite itself. The initial suite had ambiguous prompts, tests that validated implementation details instead of observable behavior, and a focus on APIs already in training data. They removed test leakage, resolved contradictions, moved to behavior-based assertions, and added tests targeting Next.js 16 APIs models haven't seen: connection(), 'use cache', cacheLife() and cacheTag(), forbidden() and unauthorized(), proxy.ts, async cookies() and headers(), plus after(), updateTag(), and refresh(). All configurations ran against the same hardened tests, with retries to rule out model variance. Most published agent evals skip that hygiene step.

Skills by default: zero improvement over baseline

Skills are an open standard for packaging domain knowledge, bundling prompts, tools, and documentation an agent can invoke on demand. There's a growing directory of them at skills.sh. For framework docs, this offers clean separation, minimal context overhead, and on-demand loading.

In practice, the skill was never invoked in 56% of eval cases. The agent had the documentation available and chose not to use it. The pass rate with the skill installed was 53%, identical to the 53% baseline with no docs at all. On the detailed breakdown, the skill configuration scored worse than baseline on tests (58% versus 63%), which suggests an unused skill sitting in the environment can add noise.

Vercel notes this isn't specific to their setup. Agents failing to reliably use available tools is a known limitation of current models. For anyone shipping skills, the content does not help when the model doesn't take the branch.

Explicit triggers got to 79%, and revealed a fragility problem

Vercel added an instruction to AGENTS.md along the lines of "explore the project structure first, then invoke the nextjs-doc skill for documentation." That pushed the trigger rate above 95% and the pass rate to 79%, a 26-point gain over baseline.

Different wordings of the trigger instruction produced dramatically different agent behavior with the same skill and the same docs:

In the 'use cache' eval, the invoke-first wording produced a correct page.tsx but missed the required next.config.ts changes. The explore-first wording got both. Small phrasing tweaks in a trigger instruction can swing task outcomes, making the mechanism brittle for production use. One prompt edit can cause a regression that requires eval coverage to catch.

Static index results

Vercel embedded a docs index directly in AGENTS.md so the agent did not need to choose whether to invoke a skill. The index is a map of where version-matched doc files live on disk, not the docs themselves. The injected content includes one instruction: "Prefer retrieval-led reasoning over pre-training-led reasoning for any Next.js tasks."

That configuration scored 100% across Build, Lint, and Test on the hardened suite.

The index keeps retrieval routing in context on every turn, instead of depending on the agent to select a skill and decide whether to read docs before or after exploring the project. Skills load asynchronously and require those sequencing choices, which created the wording fragility above.

The docs still live in a .next-docs/ directory and the agent still reads specific files when it needs them. The routing table is pinned in context, while lookups stay on demand.

Context cost

Embedding docs in AGENTS.md sounds like context bloat, and the initial injection was around 40KB. Vercel compressed it to 8KB, an 80% reduction, with no drop from the 100% pass rate. The format is a pipe-delimited index mapping documentation directory paths to the files they contain:

[Next.js Docs Index]|root: ./.next-docs
|IMPORTANT: Prefer retrieval-led reasoning over pre-training-led reasoning
|01-app/01-getting-started:{01-installation.mdx,02-project-structure.mdx,...}

8KB on every turn is a modest tax that produced a 47-point pass-rate improvement on tasks touching APIs the model has never seen. The index made retrieval reliable while adding 8KB of context.

How to set it up

For Next.js projects:

npx @next/codemod@canary agents-md

The codemod detects your Next.js version, downloads matching documentation into .next-docs/, and injects the compressed index into your AGENTS.md. Any agent that respects AGENTS.md, including Cursor, picks it up from there. Claude Code uses CLAUDE.md for the same persistent-context role, so the same pattern applies: index in the memory file, full docs on disk, and an instruction to prefer the docs over training data.

Appropriate uses for skills

Vercel does not conclude that skills are useless. Passive context won for broad framework knowledge that should inform every task the agent touches. Skills fit action-specific workflows a user explicitly triggers: "upgrade my Next.js version," "migrate to the App Router," or applying framework best practices. When the user names the action, invocation is deliberate rather than left to the model's judgment.

Skills depend on unprompted tool selection, a model capability still under active improvement. Vercel recommends compressing aggressively, structuring documentation so agents can read specific files instead of needing everything upfront, and building evals against APIs that aren't in training data. An AGENTS.md snippet gives agents a reliable path to version-matched documentation.