Edge caching rules for AI crawlers (spec)

TL;DR

• AI crawler traffic raises cache miss rates because AI bots fetch long-tail URLs that human users rarely revisit, breaking traditional locality assumptions (Cloudflare blog, 2025).
• Verify bot identity at the edge first; only then vary or branch behavior. Raw User-Agent strings are forgeable (Fastly docs).
• Set per-bot TTLs explicitly: editorial content 5-15 min for AI bots, 1-24 h for humans; static assets share the same TTL across both.
• Use Surrogate-Key headers to purge cache by content tag in ~150 ms across the global edge (Fastly purging docs).
• Honor ETag + Last-Modified. Google's crawler infrastructure explicitly uses If-None-Match and If-Modified-Since for revalidation (Google Search Central, 2024).

What this spec covers

This spec defines edge-cache behavior at the CDN layer (Cloudflare, Fastly, Akamai, CloudFront) for HTTP requests originating from AI crawlers. It complements cdn-config-ai-crawler-discoverability (which covers WAF allowlists and routing) and ai-crawler-rate-limiting-reference (which covers throttling). The goal is not to block AI bots but to ensure they receive fresh, citable content within seconds of an edit while preserving cache efficiency for human traffic. Out of scope: origin caching, browser caching, and pay-per-crawl monetization.

Four primitives are required:

1. A bot identity that survives user-agent spoofing.
2. Cache keys that route AI traffic deterministically.
3. TTLs that match content volatility per bot class.
4. A purge mechanism that invalidates the AI tier within one revalidation cycle of an edit.

Bot identity verification

Do not trust the raw User-Agent. Both Fastly and Cloudflare provide verified bot identity primitives that combine UA inspection with reverse-DNS validation:

• Cloudflare: cf.client.bot boolean and cf.verified_bot_category field via Bot Management; AI Crawl Control exposes per-operator categories such as GPTBot, ClaudeBot, PerplexityBot (Cloudflare AI Crawl Control docs).
• Fastly: client.bot.name VCL variable populated from the User-Agent header; combine with reverse-DNS or signed bot tokens for verification (Fastly client.bot.name docs).
• Akamai: Bot Manager category headers injected into the property request lifecycle.

Recommended pattern: at the edge, set a request header X-Bot-Class with one of human, search-bot, ai-bot, unverified-bot. All downstream cache logic keys off this header rather than the raw UA.

Per-bot TTL matrix

TTLs are guidance, not a contract. Adjust based on edit cadence: a daily-edited blog can extend AI TTL to 30 min; a real-time pricing API should drop to 10 s with stale-while-revalidate.

Cache key strategy: vary cautiously

Adding User-Agent to the cache key is almost always wrong. It explodes cache cardinality and reduces hit rate to near zero. Two correct alternatives:

1. Branch then merge: route AI traffic to a separate cache tier with its own TTL, but keep the cache key identical. Same URL → same object, just different freshness rules.
2. Vary on a derived class header: emit Vary: X-Bot-Class only when content actually differs by class (rare). Most sites should not vary at all.

The Vary: Accept-Encoding and Vary: Accept patterns remain standard for compression and content negotiation; do not remove them. Avoid Vary: User-Agent entirely (Fastly Vary best practices).

Revalidation primitives stay on:

• ETag: "" on every cacheable response.
• Last-Modified: on every cacheable response.
• Honor If-None-Match and If-Modified-Since with 304 Not Modified (Google Search Central, 2024).

Cloudflare Workers example

js

export default {

async fetch(request, env, ctx) {

const cf = request.cf || {};

const isVerifiedBot = cf.verifiedBotCategory != null;

const aiCategories = new Set([

"AI Crawler", "AI Search", "AI Assistant", "AI Training"

]);

const isAi = isVerifiedBot && aiCategories.has(cf.verifiedBotCategory);

const cache = caches.default;

const cacheKey = new Request(request.url, request);

let response = await cache.match(cacheKey);

if (!response) {

response = await fetch(request);

response = new Response(response.body, response);

const ttl = isAi ? 300 : 3600; // 5 min for AI, 1 h for humans

response.headers.set("Cache-Control", public, max-age=${ttl}, stale-while-revalidate=60);

response.headers.set("X-Bot-Class", isAi ? "ai-bot" : "human");

ctx.waitUntil(cache.put(cacheKey, response.clone()));

}

return response;

}

};

Fastly VCL example

vcl

sub vcl_recv {

if (client.bot.name ~ "^(GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot|Google-Extended)$") {

set req.http.X-Bot-Class = "ai-bot";

} else if (client.bot.name ~ "Googlebot|bingbot") {

set req.http.X-Bot-Class = "search-bot";

} else {

set req.http.X-Bot-Class = "human";

}

}

sub vcl_fetch {

if (req.http.X-Bot-Class == "ai-bot" && beresp.http.Content-Type ~ "text/html") {

set beresp.ttl = 300s; // AI tier freshness

set beresp.stale_while_revalidate = 60s;

} else if (beresp.http.Content-Type ~ "text/html") {

set beresp.ttl = 3600s; // human tier

}

// Surrogate keys propagated from origin are honored automatically.

}

Akamai property example (sketch)

In Akamai Property Manager, configure:

• A match condition on Bot Manager category equals AI Crawler or Generative AI.
• A behavior to set the cache TTL to 300 seconds for HTML and to forward Surrogate-Key for downstream purge.
• A separate match for Googlebot and Bingbot keeping the 1-hour HTML TTL but reusing the same cache key.

Surrogate keys for purge-on-edit

Surrogate keys allow a single API call to invalidate every cached object that shares a tag, regardless of URL. This is the cleanest way to keep the AI tier fresh without disabling caching:

Origin response:

HTTP/1.1 200 OK
Content-Type: text/html
Surrogate-Key: article:1234 section:technical author:42
Cache-Control: public, max-age=300, stale-while-revalidate=60
ETag: "v17-9f3a"
Last-Modified: Mon, 04 May 2026 02:00:00 GMT

Purge on edit (Fastly):

curl -X POST -H "Fastly-Key: $FASTLY_TOKEN" 
  https://api.fastly.com/service/$SERVICE_ID/purge/article:1234

Purge on edit (Cloudflare cache tags, Enterprise):

curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE/purge_cache" 
  -H "Authorization: Bearer $CF_TOKEN" 
  -H "Content-Type: application/json" 
  --data '{"tags":["article:1234"]}'

Fastly completes single-key purges in roughly 150 ms across the global edge (CDN Planet purge guide). Pair every edit with a tag-based purge so AI crawlers re-fetch on their next request.

Common mistakes

• Adding Vary: User-Agent globally. Cardinality explosion, near-zero hit rate, no practical benefit.
• Trusting the raw User-Agent for AI bot routing. Spoofers will inherit the AI tier or be falsely blocked.
• Setting the AI TTL to zero. Bots hammer the origin and may receive 429s, hurting citation eligibility.
• Forgetting to forward Surrogate-Key from origin. The CDN cannot purge tags it never received.
• Returning 304 without an ETag or Last-Modified set in the original 200 response, breaking revalidation.
• Using full purge instead of key purge. Triggers a thundering-herd cache miss across the edge.
• Caching robots.txt for an hour. AI access policy changes will not propagate, leading to unintended scraping.

How to validate

1. Send a request with a known AI bot UA and inspect the X-Bot-Class header in the response. It should match ai-bot.
2. Use curl -H "Fastly-Debug:1" (Fastly) or cf-cache-status (Cloudflare) to confirm hit/miss and TTL.
3. Trigger an origin edit, then request the same URL with an AI bot UA; expect a MISS followed by HIT within 1 second after the surrogate-key purge.
4. Run curl -I -H "If-None-Match: " and confirm a 304 response.
5. Inspect Cloudflare Radar or Fastly Bot Insights to verify AI bot traffic share and cache hit ratio per tier.

FAQ

Q: Should I serve different content to AI bots?

No. Different content for the same URL is cloaking and risks penalties from Google and Bing. Vary only freshness (TTL) and access (rate limits), not body content. AI engines that detect divergent content for the same URL deprioritize the source as ungroundable.

Q: Can I just disable caching for AI bots?

You can, but you should not. Origin egress costs rise sharply, response latency increases, and AI crawlers may hit rate limits more often, reducing their willingness to refetch. A 5-minute TTL with surrogate-key purge gives near-real-time freshness at minimal origin cost.

Q: How do I handle unknown AI bots?

Classify any unverified bot as unverified-bot and apply a conservative TTL plus a moderate rate limit. Do not auto-block: many small AI services run unverified pipelines that still send legitimate citation traffic. Log the user-agent, revisit weekly, and promote to ai-bot if the bot publishes a verifiable IP range or signed token.

Q: Does HTTP/2 push or 103 Early Hints help AI crawlers?

Most AI crawlers do not consume HTTP/2 push; some honor 103 Early Hints for preloads. Optimize for clean revalidation (ETag + Last-Modified) before chasing push optimizations.

Q: Should Cache-Control: no-store ever appear on AI-citable pages?

No. no-store prevents both intermediate caches and the AI engine's own re-encode pipeline from holding the response, which can drop the page from citation eligibility entirely. Use short max-age with stale-while-revalidate instead.

Q: How does this interact with pay-per-crawl?

Pay-per-crawl gates apply at request admission, before cache lookup. Once admission is granted (or the bot is allowlisted), this cache spec applies. Monetization and freshness are orthogonal layers.