Gzip vs Deflate Encoding Handshake with AI Crawlers

TL;DR

Support Brotli and gzip on text responses; deflate is rarely needed in modern stacks. Always fall back to identity when a crawler explicitly requests it. Pre-compress static assets at build time (Brotli level 11, gzip level 9), compress dynamic responses on the fly (Brotli level 4 or gzip level 6). Pair every compressed response with Vary: Accept-Encoding so caches don't serve the wrong body to a crawler.

Scope

This specification covers HTTP content encoding negotiation between AI crawlers and origin / CDN servers. Audience: performance engineers and edge-network owners.

It does not cover Transfer-Encoding (chunked) on its own or end-to-end encryption — those are orthogonal concerns.

How the handshake works

From MDN and Wikipedia's HTTP compression entry:

1. Client sends Accept-Encoding: gzip, deflate, br (with optional q-factor weights).
2. Server picks one supported encoding and compresses the response body.
3. Server replies with Content-Encoding: br and the compressed body.
4. If no acceptable encoding matches, the server returns identity (uncompressed) or 406 Not Acceptable per RFC 9110.
5. The server should add Vary: Accept-Encoding so shared caches store separate variants.

The IANA-registered tokens include gzip, compress, deflate, br, zstd, identity, and *.

Algorithm comparison

AI crawler Accept-Encoding fingerprints

Observed patterns (subject to change):

• GPTBot — typically advertises gzip, deflate in published documentation and observed traffic. Brotli support is inconsistent across crawl pools.
• ClaudeBot — typically gzip, deflate, br. Modern Brotli is generally accepted.
• PerplexityBot — typically gzip, deflate, br. Real-time fetches often look more browser-like.
• OAI-SearchBot — typically gzip, deflate, br for live retrieval.
• Googlebot / Bingbot — send gzip, deflate, br in modern crawl pools.
• Identity-only requests — some legacy or debug crawlers send Accept-Encoding: identity or omit the header. Always honor.

Log the actual Accept-Encoding per bot from your access logs; vendor patterns drift over time.

Recommended server policy

• Pre-compress static assets at build time. Generate .br (Brotli level 11) and .gz (gzip level 9) variants. Serve them when the matching token is offered.
• On-the-fly compression for HTML and API responses: Brotli level 4-6 for cacheable, gzip level 6 for highly dynamic. Higher levels increase CPU cost without proportional payload savings.
• Always emit Vary: Accept-Encoding on compressed responses. Without it, caches serve wrong-encoding bodies.
• Honor identity explicitly when offered. RFC 9110 requires falling back to identity if the client made it acceptable.
• Do not double-compress. If Content-Encoding: br is already set, do not recompress at the edge.
• Respect q-factor. Accept-Encoding: gzip;q=1.0, br;q=0.5 means prefer gzip even though br is supported.

Edge configuration patterns

Cloudflare

Enable Brotli in the dashboard or via API. Cloudflare normalizes Accept-Encoding internally and serves the best variant; preserve Vary: Accept-Encoding by default.

Fastly

Use the Auto-Compress Files feature or set beresp.gzip / beresp.brotli in VCL. Per Fastly's documentation, edge compression benefits from automatic Accept-Encoding normalization.

Nginx

gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
gzip_min_length 1024;
gzip_comp_level 6;
gzip_vary on;

brotli on;

brotli_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

brotli_comp_level 6;

brotli_static on;

The brotli_static on directive serves pre-compressed .br files when present.

Node.js (Express + compression middleware)

js

import compression from "compression"

app.use(compression({ level: 6 }))

// Add Brotli via shrink-ray-current or @fastify/compress for Fastify.

Identity fallback

When a crawler sends Accept-Encoding: identity or omits the header entirely, return the uncompressed body. RFC 9110 lets the server assume any available encoding only if identity is unavailable. Most servers handle this implicitly; verify by sending a curl request without Accept-Encoding and confirming you get an uncompressed response.

GET /article HTTP/1.1
Host: example.com
Accept-Encoding: identity

Response:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 41327

No Content-Encoding is set when the body is uncompressed.

Common pitfalls

• CDN edge transform breaks integrity hashes — a CDN that recompresses on the fly invalidates Subresource Integrity hashes embedded in HTML. Use Cache-Control: no-transform on assets where this matters.
• Wrong Content-Encoding — saying br while serving gzip causes silent decode failures. Crawlers see corrupt bodies and skip.
• Missing Vary: Accept-Encoding — cache serves a Brotli body to a gzip-only client. Crawler sees garbage.
• Compressing already-compressed assets — images, video, and .zip files don't shrink under gzip / Brotli. Skip them via MIME type filters.
• deflate ambiguity — some implementations send raw DEFLATE, others send zlib-wrapped. Modern advice: skip deflate, ship gzip.
• Responding 406 Not Acceptable — if no compression matches, return uncompressed identity instead of 406. Crawlers do not retry 406s.

FAQ

Q: Should I serve Brotli or gzip to AI crawlers?

Brotli when the crawler's Accept-Encoding includes br and you've pre-compressed the asset (best ratio). Gzip when only gzip is offered or for dynamic responses where Brotli's higher CPU cost matters.

Q: Is deflate still useful?

Rarely. Historical implementation inconsistency between raw DEFLATE and zlib means some clients fail to decode. Ship gzip and Brotli; skip deflate for new deployments.

Q: What happens if no Accept-Encoding matches?

Return the body uncompressed with no Content-Encoding. Per RFC 9110, identity is the implicit fallback. Returning 406 confuses crawlers and they typically do not retry.

Q: Do I need Vary: Accept-Encoding?

Yes whenever you serve compressed responses through a shared cache. Without it, the cache may serve a Brotli body to a gzip-only client.

Q: How aggressive should my compression level be?

Level 11 (Brotli) and 9 (gzip) for pre-compressed static assets at build time. Level 4-6 for dynamic responses where CPU cost matters. Higher levels rarely produce meaningful savings on text content beyond level 6 for gzip.

Q: Will compressing responses slow down crawlers?

No. Compressed bodies are smaller and decode quickly. Brotli decompression speed is comparable to gzip per public benchmarks. Crawlers fetch faster, which improves their effective per-host throughput.