Cache-Control Headers Reference for AI Crawlers

TL;DR

Return Cache-Control: public, max-age=600, s-maxage=3600, stale-while-revalidate=86400 on canonical content pages, Cache-Control: public, max-age=31536000, immutable on hashed static assets, and Cache-Control: no-store on user-specific or auth-bound responses. Add ETag (preferred by Google) for revalidation and Last-Modified as a fallback.

Scope

This reference lists the RFC 9111 Cache-Control directives that influence how AI crawlers like GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot refresh content. It applies to origin servers and CDN edge caches.

It does not cover service-worker caching or browser cache eviction strategies.

How AI crawlers use cache headers

Google confirms that its crawler infrastructure supports heuristic HTTP caching via ETag / If-None-Match and Last-Modified / If-Modified-Since (Google Search Central, 2024). When the origin returns 304 Not Modified, the crawler reuses the cached body and saves bandwidth on both sides. AI crawlers operated by OpenAI, Anthropic, and Perplexity follow the same pattern in practice.

Real-world data: Cloudflare reports significant bandwidth pressure from AI crawlers, with public examples like Wikimedia observing a 50% surge in multimedia bandwidth tied to bulk image scraping (Cloudflare, 2025). Tighter cache directives plus ETag revalidation directly reduce that pressure.

Directive matrix

Directive	Effect	AI crawler note
public	Response is shareable across users and CDNs.	Required for shared CDN caches that front AI crawlers.
private	Response is for a single user; only the browser may cache.	Hides response from CDNs; AI crawlers receive it but cannot reuse.
max-age=N	Fresh for N seconds for client caches.	Sets minimum revisit floor for crawlers without aggressive heuristics.
s-maxage=N	Fresh for N seconds for shared caches; overrides max-age.	Lets you tune CDN cache independently from browser.
no-cache	Cached but must revalidate before reuse.	Forces ETag round-trip; safe and bandwidth-friendly.
no-store	Must not cache.	Tells AI crawlers not to retain the response. Use only for sensitive data.
must-revalidate	After expiry, must revalidate before serving stale.	Caches that respect this won't serve stale to crawlers.
immutable	Body never changes during freshness window.	Use on hashed static assets (JS/CSS/images with content hash).
stale-while-revalidate=N	Serve stale up to N seconds while async revalidating (RFC 5861).	Keeps AI surfaces fast during cache misses.
stale-if-error=N	Serve stale up to N seconds when origin errors.	Prevents crawler 5xx loops; supported by major CDNs.
no-transform	Intermediaries must not modify content.	Prevents CDN image recompression that breaks integrity hashes.

Recipes by content type

• Canonical article / doc page — public, max-age=600, s-maxage=3600, stale-while-revalidate=86400 plus ETag. Crawlers can re-fetch hourly via revalidation; cheap when content is unchanged.
• Hashed static assets (JS/CSS bundles with content hash) — public, max-age=31536000, immutable. Crawlers fetch once and never revalidate.
• Logged-in dashboards — private, no-store. Crawlers should not retain or share these responses.
• API responses — public, max-age=60 if cacheable; pair with proper Vary to avoid cache poisoning across user agents.
• Sitemap / robots.txt — public, max-age=300, stale-while-revalidate=3600. Allows fast iteration when crawl directives change.
• 404 pages — public, max-age=60. Avoid long caching; allows quick correction.

ETag and Last-Modified pairing

Google recommends ETag over Last-Modified because it avoids HTTP-date parsing edge cases (Google Search Central, 2024). Best practice:

• Always include ETag on cacheable HTML and asset responses.
• Strong validators (e.g., "abc123") for byte-identical content; weak validators (W/"abc123") for semantically-identical responses.
• Include Last-Modified as a fallback for crawlers that don't surface ETag handling.
• Implement If-None-Match / If-Modified-Since on the server so revalidation returns 304.

Surrogate-Control for CDN-only caching

Use Surrogate-Control (Fastly) or Cache-Tag / CDN-Cache-Control to tune CDN cache lifetime independently from Cache-Control. This lets you cache at the edge for hours while telling browsers and AI crawlers to revalidate every few minutes:

Cache-Control: public, max-age=60
CDN-Cache-Control: public, max-age=3600
Surrogate-Control: max-age=3600, stale-while-revalidate=86400

Cache-busting strategies

• Hashed asset URLs for JS/CSS/images so immutable is safe (/static/app.abc123.js).
• Versioned query strings for content where you can't change the URL but need to invalidate (?v=2026-05-03).
• Surrogate-key purge at the CDN to invalidate specific tags after content publish.
• Clear-Site-Data: cache for hard reset on auth events; ignored by most AI crawlers.

Common pitfalls

• no-cache, no-store, must-revalidate everywhere — wastes bandwidth, slows AI crawlers, and trains them to deprioritize the host.
• Long max-age on canonical articles — AI surfaces can serve stale answers for days.
• Missing ETag — every revisit becomes a full body transfer.
• private on shareable content — hides response from edge caches that front AI crawlers.
• Conflicting Cache-Control and Expires — Cache-Control wins per RFC 9111; remove the Expires header to avoid confusion.
• no-transform omitted on image-heavy pages — CDN recompression can break image hashes embedded in structured data.

FAQ

Q: Do AI crawlers actually honor Cache-Control?

Google's crawlers support heuristic HTTP caching via ETag and Last-Modified per their public documentation. OpenAI's GPTBot, Anthropic's ClaudeBot, and PerplexityBot follow standard HTTP caching semantics in practice, including 304 Not Modified handling.

Q: Should I prefer ETag or Last-Modified?

ETag, per Google's December 2024 guidance. ETag avoids HTTP-date format issues and supports both strong and weak validators. Keep Last-Modified as a fallback.

Q: What does immutable actually do?

It tells caches the body will never change while the response is fresh, so the cache can skip revalidation entirely. Use it only on URLs that contain a content hash and will never change in place.

Q: Will stale-while-revalidate confuse AI crawlers?

No. Crawlers fetch the same response a browser would; the directive is honored by the cache layer and is invisible to the client. The benefit is that AI surfaces stay fast during background revalidation.

Q: Can I use no-store to opt out of AI training?

Not reliably. no-store instructs caches not to retain the response, but training crawlers download and process the body before any cache layer sees it. Use robots.txt or X-Robots-Tag for opt-out signals.

Q: How does Cache-Control interact with the Vary header?

Vary tells caches which request headers cause different responses. If you serve different content per User-Agent, you must include Vary: User-Agent so caches don't serve a bot response to a human user (or vice versa).