Robots.txt vs ai.txt vs llms.txt: Practical Comparison

TL;DR

robots.txt, ai.txt, and llms.txt sit at three different layers. robots.txt is the only one that actually controls crawler access; ai.txt signals AI training and licensing preferences; llms.txt publishes a curated AI-readable site map for compliant retrievers. Most production sites should publish all three rather than treat them as alternatives.

Quick verdict

• Use robots.txt to allow or block named AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). It is the only one of the three that actually controls access.
• Use ai.txt to publish AI-training and content-licensing preferences in a machine-readable form.
• Use llms.txt to give compliant AI retrievers a clean, curated map of your highest-citation-value pages.

These are layers, not alternatives.

Key differences

What each file actually does

robots.txt

The original 1994 Robots Exclusion Protocol. It tells named user-agents which paths they may or may not crawl. For AI search, the relevant user-agents include:

• GPTBot (OpenAI training)
• OAI-SearchBot (ChatGPT search retrieval)
• ChatGPT-User (ChatGPT browsing on user request)
• ClaudeBot (Anthropic training)
• Claude-User (Claude browsing)
• PerplexityBot (Perplexity retrieval)
• Google-Extended (Google AI training opt-out token)
• CCBot (Common Crawl)

Blocking GPTBot while allowing OAI-SearchBot is the standard pattern for sites that want to be cited but not used in pre-training.

ai.txt

Proposed by Spawning, ai.txt expresses content licensing and AI-training preferences. Categories include text, images, audio, and video, each marked allow / disallow. It does not control access; it states intent. Adoption is concentrated in publishing and creative-asset sites.

llms.txt

Proposed by Answer.AI, llms.txt is a Markdown index of your most important pages designed to be read by LLMs at retrieval time. It is purely advisory and does not block anything. See the full llms.txt specification.

When to use each

When to use robots.txt

Always. Every site should publish robots.txt, even if it is empty (Allow: /). Use it specifically when you want to:

• Block a specific AI bot from training while allowing search retrieval.
• Disallow paths that have no value being indexed (logins, faceted search, drafts).
• Apply different rules to different bots.

When to use ai.txt

Use when you have a defensible content-licensing position to publish, especially for:

• Editorial publishers protecting copy.
• Stock-asset libraries protecting images.
• Brands that want a public, machine-readable AI-training stance.

ai.txt does not enforce; it documents intent. Pair it with robots.txt for actual blocking.

When to use llms.txt

Use when you want AI engines to cite the right pages from your site. Documentation, knowledge bases, and reference-heavy domains benefit most. Adoption is voluntary but growing; the cost to publish is low and the upside is targeted citation control.

How they fit together

[ Edge / WAF / rate limit ] <- enforcement, blocks misbehaving bots

[ robots.txt ] <- advisory access rules per user-agent

[ ai.txt ] <- training/licensing preference signal

[ llms.txt ] <- curated retrieval map for compliant LLMs

[ sitemap.xml ] <- exhaustive URL discovery for search

For real enforcement (DDoS, scraping abuse), see AI crawler rate limiting.

Common misconceptions

• "llms.txt replaces robots.txt." No. Different layer.
• "ai.txt blocks AI training." It does not enforce. Use robots.txt to disallow training bots.
• "If I have llms.txt I don't need a sitemap." sitemap.xml remains the source of truth for indexing exhaustively.
• "All AI bots respect these files." Most well-known ones do. Long-tail scrapers do not. Use rate limiting at the edge for actual control.

Implementation checklist

• [ ] Publish robots.txt with explicit rules for major AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot).
• [ ] Decide your training stance and publish ai.txt to advertise it.
• [ ] Author llms.txt with curated, citation-worthy pages.
• [ ] Verify all three return a 200 with the right Content-Type at the root path.
• [ ] Re-check after every release.
• [ ] Monitor logs for AI user-agents quarterly.
• [ ] Combine with edge rate limiting for misbehaving crawlers.

FAQ

Q: Do I need all three files?

Most mature sites should have robots.txt and llms.txt. ai.txt is optional and most useful when you have a publishable AI-training stance.

Q: Will blocking GPTBot stop ChatGPT from citing me?

It may. OpenAI separates GPTBot (training) from OAI-SearchBot and ChatGPT-User (retrieval and on-demand browsing). Block training while allowing the retrieval agents if you want citations without training contributions.

Q: Does robots.txt actually block bad actors?

No. It is an honor system. For real enforcement use WAF rules, rate limits, and challenge pages. See AI crawler rate limiting.

Q: Can these files conflict?

They can. If robots.txt disallows a path that llms.txt links to, compliant retrievers will skip the link. Audit alignment whenever you change either file.

Q: How often should I review my AI bot rules?

Quarterly at minimum. Vendors add and rename user-agents frequently; old robots.txt files often miss the latest bots.