AI Crawl Budget: Controlling What LLMs Index

AI crawl budget is the finite attention AI bots spend on a domain per cycle. Sites can steer that budget toward high-value, citation-worthy pages by tightening robots.txt, suppressing low-value URLs, publishing llms.txt, and keeping internal links aligned with editorial priority.

TL;DR

AI crawlers do not visit every page on a site every day. A finite budget gets spent per crawler, per cycle, and you control where it lands. Block noise (faceted URLs, drafts, infinite calendars), advertise priority via llms.txt, keep internal links pointing at the pages you want cited. Accept that training and retrieval bots need different policies.

What is AI crawl budget?

"Crawl budget" originally described how Googlebot allocates fetches across a domain. AI crawl budget extends the idea to AI-specific user-agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, and others. Each has its own crawl behavior, rate, and decision logic about which URLs to fetch.

A budget is constrained by:

Server response time and uptime.
Vendor-side rate ceilings.
Path-level rules in robots.txt.
Discovery surface (sitemap, llms.txt, internal links, external links).

If the budget gets spent on low-value URLs, citation-worthy pages may go uncrawled, untrained, and uncited.

Why it matters for AI citations

Coverage gap. Pages that are never crawled cannot be cited. Period.
Freshness gap. A high-traffic article that is not recrawled after an update will continue to be cited with stale facts.
Embedding cost. AI engines often re-embed crawled content. Wasted crawls mean wasted embeddings on pages you do not want surfaced.
Citation share leakage. When ranked alongside competitors, the page the engine actually loaded wins. If your important page was not loaded this cycle, you lose.

How AI crawl budget works

AI crawlers behave more like search bots than human visitors. The simplified loop:

Discover URLs from sitemap.xml, llms.txt, internal and external links, prior crawl data.
Filter disallowed paths via robots.txt.
Schedule within the per-domain rate ceiling.
Fetch with the bot's user-agent and Accept-Encoding negotiation.
Score the page (content-type, freshness, authority signals).
Persist into the model's retrieval index, training corpus, or both.
Decide whether to recrawl, and how soon.

Different vendors weight these steps differently. Perplexity tends to recrawl frequently for retrieval; OpenAI's GPTBot historically crawls less often but more deeply for training.

Training bots vs retrieval bots

Category	Examples	Citation impact	Frequency
Training	GPTBot, ClaudeBot, Google-Extended, CCBot	Indirect (via model knowledge)	Episodic
Retrieval	OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User	Direct (live citations)	Frequent

Treat them differently. A retrieval bot fetching every product page weekly is signal; a training bot scraping every faceted URL nightly is waste. Pair this with robots.txt vs ai.txt vs llms.txt to set the right policy per category.

Tier-based allocation matrix

Assign each URL to a tier and steer crawl behavior accordingly:

Tier	Examples	Robots.txt	llms.txt	Sitemap	Internal links
T1 — Citation-critical	Pillar guides, references, glossaries	Allow all retrieval bots	Listed	Listed, high priority	Hub-linked
T2 — Supportive	Tutorials, comparisons, blog	Allow retrieval, allow training	Optionally listed	Listed	Cross-linked
T3 — Tactical	Case studies, news posts	Allow	Not listed	Listed	Light
T4 — Operational	Login, search, admin, faceted	Disallow all bots	Not listed	Not listed	Nofollow internally
T5 — Ephemeral	Drafts, staging, A/B variants	Disallow + noindex	Not listed	Not listed	None

How to control your AI crawl budget

1. Audit current crawl behavior

Filter the access log by user-agent and report:

Total requests per AI bot per day.
Top-fetched paths.
Bytes transferred.
4xx and 5xx ratios.

Look for waste: duplicate URLs, parameter explosions, archive pagination, internal search results.

2. Tighten robots.txt

Disallow Tier 4 and 5 paths for all AI bots.
Where appropriate, disallow training bots while allowing retrieval bots (e.g. block GPTBot, allow OAI-SearchBot).
Keep the file small; do not list every URL.

3. Publish llms.txt

List only Tier 1 and high-priority Tier 2 pages. See the llms.txt specification. Treat it as the editorial bot menu.

4. Clean the sitemap

Remove non-canonical URLs.
Mark lastmod accurately so retrieval bots know when to recrawl.
Split into segmented sitemaps if larger than 50,000 URLs.

5. Fix internal-link gravity

AI bots follow internal links. If your nav and homepage point at Tier 4 pages (login, account), bots burn budget there. Audit:

Top 50 internally linked pages — should match Tier 1.
Orphaned Tier 1 pages — fix or kill.
Faceted URLs in pagination — add rel=nofollow or block via robots.txt.

6. Improve server economics

Faster TTFB and lower error rates often raise the per-domain ceiling vendors apply. Ensure:

Cache-Control headers are set on Tier 1 pages.
Compression (Brotli, gzip) is enabled.
5xx ratios stay below 1%.

7. Monitor citation outcomes

Use LLM citation benchmarks to confirm that crawl-budget changes actually moved citations. Crawl is a means; citations are the KPI.

Common mistakes

Treating all AI bots the same. Training and retrieval need different rules.
Blocking retrieval bots by accident. A blanket block of "AI" can cut you off from ChatGPT search citations.
Letting search facets explode. A handful of filter combinations can balloon into millions of URLs that bots will dutifully crawl.
Ignoring 5xx noise. Vendors throttle aggressively when error rates spike.
Stale lastmod values. Retrieval bots rely on this to recrawl freshness-sensitive pages.

Validation

AI bot logs show 80%+ of fetches landing on Tier 1-T2 URLs.
4xx and 5xx ratios for AI bots remain under 2%.
New Tier 1 pages appear in citation benchmarks within two crawl cycles.
Disallowed paths show zero successful fetches from compliant bots.
llms.txt and sitemap.xml stay in sync after every release.

FAQ

Q: Do AI crawlers honor crawl-delay in robots.txt?

Most major AI bots (GPTBot, ClaudeBot) document support for crawl-delay, though enforcement varies. Treat it as advisory and rely on edge rate limits for hard ceilings.

Q: How do I see what GPTBot or PerplexityBot actually crawled?

Filter access logs by user-agent. CDN dashboards (Cloudflare, Fastly) usually surface AI bot categories directly. Cross-reference with sitemap and internal-link priority to find leakage.

Q: Will publishing llms.txt reduce my crawl budget waste?

It does not lower fetches in absolute terms, but for compliant retrievers it improves allocation: more fetches land on the curated list and fewer on noise. Pair it with robots.txt to actually shrink the addressable surface.

Q: What if a vendor crawls aggressively?

Set a crawl-delay, then enforce with edge rate limits if compliance is poor. See AI Crawler Rate Limiting. Blocking entirely is usually counterproductive for citation goals.

Q: Should new content go in llms.txt immediately?

Only when the page is editorially complete. llms.txt is a curated list, not a publishing pipeline. Add new entries on each content release once a page is genuinely Tier 1.