llms.txt is a curated, machine-readable content menu retrieval bots use to find your best material. ai.txt is a permissions ledger expressing training opt-outs. Both ride alongside robots.txt without replacing it. Implement llms.txt first, then add ai.txt for governance.

TL;DR

They solve different problems. llms.txt advertises what to read; ai.txt advertises what may be used for training. Robots.txt remains the access-control mechanism for compliant bots. Most sites should publish all three, in that order of priority. Validate that each file returns 200 with correct MIME, is referenced from your headers or homepage where appropriate, and is mirrored across staging.

Why this comparison exists

The ecosystem of "AI directive files" is genuinely confusing because the proposals come from different communities and overlap in scope:

llms.txt — community proposal at llmstxt.org for a curated content index aimed at LLMs.
ai.txt — Spawning's proposal at site.spawning.ai/ai-txt for declaring AI training permissions per data type.
robots.txt — the long-standing access-control file (RFC 9309).

Without a clear comparison, teams ship inconsistent or conflicting files and sometimes block their own retrieval bots. See the broader robots.txt vs ai.txt vs llms.txt reference for context.

Side-by-side specification

Dimension	llms.txt	ai.txt
Goal	Curated content menu for LLMs	Training permissions ledger
Authoring style	Markdown links with descriptions	Permission directives per data type
Path	/llms.txt (root)	/ai.txt (root)
MIME type	text/markdown	text/plain
Audience	Retrieval bots & RAG agents	Crawler operators (training)
Status	Community proposal, growing adoption	Industry proposal (Spawning), partial adoption
Replaces robots.txt?	No	No
Compatible with	sitemap.xml, schema.org	robots.txt, DSA / EU AI Act stances
Caching	Public, 1-7 days typical	Public, 1-7 days typical
Risk if misconfigured	Bots crawl noise	Training opt-out ignored

File examples

llms.txt

Geodocs

Practical guides for getting cited by AI search.

Pillars

GEO playbook: start here for end-to-end strategy.
AI Overviews optimization: pre-publish through measurement checklist.

References

Citation hygiene: source selection rules.

ai.txt

Spawning ai.txt — Geodocs

User-Agent: *

Disallow: /private/

Disallow: /staging/

Training data declarations

Dataset: text

Permission: opt-in

Contact: licensing@geodocs.dev

Dataset: image

Permission: opt-out

Hosting and headers

Serve from the canonical root: https://yourdomain.com/llms.txt. Subdomains need their own copy.
Set Content-Type: text/markdown; charset=utf-8 for llms.txt and text/plain; charset=utf-8 for ai.txt.
Allow caching: Cache-Control: public, max-age=86400.
Avoid placing them behind authentication, CAPTCHA walls, or aggressive bot protection rules. Many WAF rules silently block AI bots; whitelist the relevant user-agents.
For Cloudflare and similar edges, exempt these paths from "Block AI Bots" rules.

Decision matrix

When to publish:

llms.txt — always, once you have at least 5 Tier-1 pages. Low cost, immediate steering.
ai.txt — yes, if you care about training-time opt-outs or jurisdictional disclosure (EU AI Act, DSA).
robots.txt — always. Keep it authoritative for access control.

When not to publish:

Skip llms.txt if your editorial graph is unstable; a list pointing at deleted URLs is worse than no list.
Skip ai.txt only if your governance posture is fully open and you have legal sign-off.

Fail-safe rollout plan

Inventory. List Tier-1 URLs with descriptions.
Draft llms.txt locally; validate Markdown syntax.
Stage. Publish under /llms.txt on a staging subdomain that is not crawled.
Smoke-test with curl -I and curl /llms.txt to verify MIME and content.
Promote. Push to production and confirm the file is reachable from a clean profile.
Monitor. Watch retrieval-bot logs for 200s on /llms.txt.
Iterate. Add ai.txt next, repeating the same procedure.
Document in your CMS that any Tier-1 page change must trigger an llms.txt re-build.

Validation

curl -I https://yourdomain.com/llms.txt returns 200 with correct MIME.
The same for /ai.txt.
Listed URLs return 200 (not 301, 404, or canonicalized away).
robots.txt does not disallow /llms.txt or /ai.txt.
Edge rules (Cloudflare, Akamai) do not require challenge for compliant bots.
Diff the production file against the source-of-truth list per release.

Common mistakes

Treating ai.txt as access control. It is not enforced; robots.txt and edge rules enforce.
Listing every URL in llms.txt. Curated, not exhaustive. List Tier-1 only.
Forgetting subdomain copies. docs.example.com needs its own llms.txt.
Stale lists. Pages move; broken links erode trust signals.
Mixing the formats. llms.txt is Markdown; ai.txt is plain text directives. Don't conflate.

FAQ

Q: Will publishing llms.txt slow down my site?

No. It is a small static file served once per crawler cycle. Cache headers keep cost negligible.

Q: Do AI bots actually read these files yet?

llms.txt is read by a growing set of retrieval agents and RAG pipelines, including some commercial AI crawlers. ai.txt has narrower adoption — Spawning's ecosystem and a few crawlers honor it. Both are forward-looking moves with low downside.

Q: What's the relationship to llms-full.txt?

llms-full.txt is an optional companion proposal containing the full bodies of listed pages. Treat it as a Tier-1 mirror that AI agents can ingest without crawling each URL. Adopt it after llms.txt is stable.

Q: Should I block AI training while keeping retrieval allowed?

That is a common posture. Use ai.txt to declare training opt-out, and use robots.txt to allow retrieval-only user-agents (e.g. OAI-SearchBot, PerplexityBot) while blocking training user-agents (e.g. GPTBot, CCBot). See AI Crawl Budget for tier-based steering.

Q: Do I need legal sign-off?

For ai.txt, yes — it is effectively a public statement of licensing intent. For llms.txt, generally no, but coordinate with the editorial owner so the curated list reflects the brand.

LLMs.txt vs AI.txt: Practical Implementation Comparison