Vary Header Patterns for AI Crawlers

TL;DR

Default to Vary: Accept-Encoding on every cacheable response. Add Vary: Accept-Language only when content negotiation by language is real. Avoid Vary: User-Agent on shared caches because there are thousands of crawler and browser variants — use surrogate keys for tag-based invalidation instead. Always send the same Vary value on 200, 304, and the default response per MDN.

Scope

This specification covers the Vary response header for sites serving content to AI crawlers and human users via shared caches (CDNs, edge proxies). Audience: edge-network engineers and platform leads.

It does not cover client-side cache behavior in browsers; that is well documented in MDN.

How Vary works

A cache stores responses keyed by URL. When the response includes Vary: Accept-Encoding, the cache stores a separate copy for each unique Accept-Encoding request value. The next request with the same encoding hits the cache; a request with a different encoding misses.

For AI crawlers, the practical implications are:

• Vary: Accept-Encoding lets the CDN store both compressed and uncompressed copies, serving each crawler the right body.
• Vary: User-Agent creates a separate cache entry for every variant of every user agent string. CDNs see many thousands of distinct UA strings per day, so the cache becomes nearly useless (HTTP Archive Almanac, 2020).
• Missing Vary on negotiated content can leak a bot response to a human user (or vice versa).

Recommended values

Anti-patterns to avoid

Vary: *

A wildcard Vary instructs caches to never reuse the response. Use Cache-Control: no-store instead if that is the intent; mixing them confuses cache implementations.

Vary: User-Agent on shared caches

CDNs see thousands of UA strings per day. Each unique UA becomes a separate cache entry. Akamai documentation explicitly warns that they will not cache responses with Vary values other than Accept-Encoding to prevent fragmentation. Cloudflare and Fastly behave similarly by default: a Vary: User-Agent response is often treated as uncacheable.

If you genuinely need bot-specific markup, do it at the edge with surrogate keys and rewrite the cache key explicitly.

Inconsistent Vary across responses

Per MDN, the same Vary value must appear on 200, 304, and any default response for the URL. A 200 with Vary: Accept-Encoding followed by a 304 with no Vary causes some caches to drop the entry, defeating revalidation entirely.

Forgetting to include Vary when content actually varies

If you serve different markup based on a header but omit Vary, downstream caches will serve one variant to all users. Bots may see a human-targeted response with personalization placeholders, which can lower citation quality.

Surrogate keys as a better alternative

Where you would reach for Vary: User-Agent, prefer surrogate keys at the CDN. Per Fastly's documentation, surrogate keys (and equivalents like Cloudflare Cache Tags or Vercel cache tags) tag cached entries by content type. This allows tag-based purging without fragmenting the cache by user-agent variant.

A representative pattern:

Cache-Control: public, max-age=3600
Vary: Accept-Encoding
Surrogate-Key: article-1234 article-list product-42

When content updates, purge by surrogate key; the cache replaces the affected entries without fragmenting the cache layout.

Per-CDN behavior

• Cloudflare — Custom cache keys via Cache Rules let you derive cache identity from explicit headers or cookies, replacing Vary: User-Agent patterns (Cloudflare, 2025).
• Fastly — Use req.http.X-AI-Bot set in vcl_recv based on User-Agent, then include the variable in the cache key. Surrogate keys handle invalidation.
• AWS CloudFront — Cache policies expose specific headers as cache key components. Avoid forwarding User-Agent; whitelist only the headers you actually negotiate on.
• Vercel — Use Vary alongside Edge Cache and tag-based revalidation rather than User-Agent variance.

Examples

Standard content page

HTTP/1.1 200 OK
Cache-Control: public, max-age=600, s-maxage=3600, stale-while-revalidate=86400
Vary: Accept-Encoding
Content-Encoding: br

Multi-language API

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: public, max-age=300
Vary: Accept, Accept-Encoding, Accept-Language
Content-Language: en-US
Content-Encoding: gzip

Bot-specific markup at the edge

HTTP/1.1 200 OK
Cache-Control: public, max-age=3600
Vary: Accept-Encoding
Surrogate-Key: bot-variant article-1234
X-AI-Bot-Variant: enabled

The edge worker computes a custom cache key that includes a derived bot flag without exposing the full User-Agent to the cache.

Common pitfalls

• Mismatch between Vary and CORS — Forgetting Vary: Origin on CORS responses leaks one origin's headers to another.
• Different Vary on 200 and 304 — Causes cache eviction; ensure your origin sends consistent values.
• Vary: Cookie — Catastrophic for shared caches. Cookies vary per-user; the cache will never be reused.
• Sending Vary on uncacheable responses — Harmless but pointless. Reserve Vary for Cache-Control: public responses.
• Using Vary: User-Agent to differentiate AMP — AMP variants belong on different URLs (?amp=1 or /amp/).
• CDN strips Vary — Some CDNs rewrite Vary to remove fields they don't honor. Verify with curl -I from outside your network.

FAQ

Q: Should I always include Vary: Accept-Encoding?

Yes when you serve compressed content. Without it, caches may serve a brotli-encoded body to a client that only accepts gzip.

Q: Why is Vary: User-Agent considered an anti-pattern?

Because shared caches see many thousands of distinct UA strings, each producing its own cache entry. The cache becomes too fragmented to serve repeated requests, and many CDNs simply mark the response uncacheable instead.

Q: How do I serve bot-specific markup without Vary: User-Agent?

Detect the bot at the edge, set a custom cache key based on a derived flag (e.g., bot=1 or bot=0), and store two variants. Surrogate keys handle invalidation.

Q: Is Vary: * the same as no caching?

Functionally yes for most caches. Use Cache-Control: no-store instead for clarity; mixing them produces inconsistent behavior.

Q: Does Vary apply to 304 Not Modified responses?

Yes. Per MDN, the Vary header value must match across 200 and 304 responses for the same URL or some caches will discard the entry.

Q: How do AI crawlers handle Vary?

Like any HTTP client. They send their request headers; the CDN serves the matching cache variant. The crawler does not need to know about Vary directly — the cache layer does the work.