Geodocs.dev

HTTP/2 vs HTTP/3 for AI Crawlers

ShareLinkedIn

Open this article in your favorite AI assistant for deeper analysis, summaries, or follow-up questions.

HTTP/3 over QUIC reduces head-of-line blocking and connection-establishment latency compared to HTTP/2 over TCP, but as of 2026 the major AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and Applebot-Extended — still negotiate HTTP/2 by default and fall back to HTTP/1.1 on origins that lack ALPN h2 support.

TL;DR

For visitor-facing traffic, enable HTTP/3 (QUIC). For AI crawler traffic, the practical decision is to keep HTTP/2 reliable and well-tuned, because most AI bots do not yet upgrade to QUIC even when servers advertise Alt-Svc. Continue serving HTTP/3 from the edge so browsers benefit, but do not break HTTP/2 origin support to chase QUIC for crawlers — the gain is theoretical and the fallback cost is real.

Quick verdict

QuestionRecommendation
Should I enable HTTP/3 site-wide?Yes, for human visitors. CDN-level toggle on Cloudflare, Fastly, or Vercel.
Will AI crawlers use HTTP/3 if available?Mostly no. Bots advertise h2 and http/1.1 in ALPN; QUIC adoption among bots remains low.
Should I disable HTTP/2 for crawlers?No. HTTP/2 is the de facto baseline; disabling it forces HTTP/1.1 with worse multiplexing.
Does HTTP/3 affect indexing or citation?No direct signal. Crawl efficiency and freshness are determined by lastmod, robots, and server response time — not transport version.

Key differences: HTTP/2 vs HTTP/3

DimensionHTTP/2HTTP/3
TransportTCP + TLS 1.2/1.3QUIC over UDP (TLS 1.3 built-in)
HandshakeTCP 3-way + TLS = 2-3 RTTQUIC = 1 RTT, 0-RTT for resumed
Head-of-line blockingAt TCP layer (one lost packet stalls all streams)Eliminated at transport — streams independent
MultiplexingYes (per-connection streams)Yes (per-connection streams, no TCP HoL)
Header compressionHPACKQPACK (decoupled from stream order)
Connection migrationNo (breaks on IP change)Yes (Connection ID survives network swap)
Middlebox compatibilityHigh (TCP is universal)Variable (UDP often filtered or rate-limited)
Default port443/TCP443/UDP
NegotiationALPN over TLSAlt-Svc header advertises h3 endpoint
Server CPU costLower (kernel TCP stack)Higher (userspace QUIC stack)

Cloudflare reports global HTTP/3 share at roughly 30-35% of web requests in 2025, with HTTP/2 still dominant. Crucially, Cloudflare's own multi-year analysis found that "search engine indexing and social media bots continued to effectively ignore the latest version of the web's core protocol" — the user-centric latency gains of QUIC offer minimal benefit to asynchronous crawlers.

AI crawler protocol support matrix

The following matrix reflects observed behavior in production CDN logs and published documentation as of early 2026. AI crawler vendors do not formally publish HTTP/3 support statements; this table is based on user-agent ALPN negotiation patterns from Cloudflare, Fastly, and origin server logs.

Crawler (vendor)HTTP/1.1HTTP/2HTTP/3 (QUIC)Notes
GPTBot (OpenAI)YesYes (default)Not observedFalls back to HTTP/1.1 if no h2
OAI-SearchBot (OpenAI)YesYes (default)Not observedPowers ChatGPT search citations
ChatGPT-User (OpenAI)YesYesNot observedOn-demand fetch for user prompts
ClaudeBot (Anthropic)YesYes (default)Not observedHigh-volume training crawler
Claude-Web (Anthropic)YesYesNot observedOn-demand fetch for Claude.ai
PerplexityBot (Perplexity)YesYes (default)Not observedCitation-focused crawler
Perplexity-User (Perplexity)YesYesNot observedReal-time user query fetch
Google-Extended (Google)YesYes (default)Inherits Googlebot stack — limited h3Opt-out for Bard/Gemini training
Googlebot (Google)YesYes (default)Limited regional h3Long-tail h2 still dominant
Applebot-Extended (Apple)YesYes (default)Not observedPowers Apple Intelligence training
Bytespider (ByteDance)YesYesNot observedAggressive crawl rate
CCBot (Common Crawl)YesPartialNot observedMany requests still HTTP/1.1

The consistent pattern: AI bots advertise h2,http/1.1 in ALPN and ignore Alt-Svc: h3=":443" headers. This matches the broader observation that bots are designed for asynchronous bulk crawl, where 100ms saved on handshake is irrelevant to throughput.

Why AI crawlers lag on HTTP/3 adoption

Three practical reasons explain the gap:

  1. No latency win for batch crawlers. QUIC's 0-RTT and 1-RTT handshakes shave hundreds of milliseconds off cold connections — a meaningful gain for browsers fetching above-the-fold content but negligible for crawlers that process millions of URLs in async pipelines.
  2. UDP middlebox issues. Many enterprise networks rate-limit or filter UDP/443. Crawlers running across diverse network paths prefer the universal compatibility of TCP.
  3. Userspace QUIC cost. QUIC implementations run in userspace and consume more CPU per connection than kernel TCP. At crawler scale (millions of requests per hour), the CPU overhead matters more than per-connection latency.

Performance benchmarks: when HTTP/3 actually helps

Real-world benchmarks show HTTP/3's advantage is conditional, not absolute:

  • Lossy mobile networks (3G/4G with packet loss): HTTP/3 reduces page load time by 30-55% versus HTTP/2 in scenarios with ~15% packet loss.
  • Small payloads on stable networks: Cloudflare measured HTTP/3 at 443ms vs HTTP/2 at 458ms for a 15 KB page — a 3% edge.
  • Larger payloads (1 MB+) on stable networks: HTTP/3 was slightly slower than tuned HTTP/2 with BBR congestion control (2.33s vs 2.30s).
  • Local/low-latency benchmarks: Some practitioners report HTTP/3 being 50-100x slower on local networks because of userspace stack overhead at near-zero latency.

For AI crawlers, which fetch from datacenters with stable low-loss network paths, HTTP/2 over TCP with BBR or CUBIC congestion control usually outperforms HTTP/3.

CDN configuration recommendations

Cloudflare

HTTP/3 is on by default for free-tier zones; paid plans expose a toggle in Speed → Optimization → Protocol Optimization. To enable via API: PATCH the http3 zone setting to on. Keep HTTP/2 enabled simultaneously — Cloudflare negotiates per-client.

Cloudflare API example

curl -X PATCH "https://api.cloudflare.com/client/v4/zones/{zone_id}/settings/http3"

-H "Authorization: Bearer $CF_TOKEN"

-H "Content-Type: application/json"

--data '{"value":"on"}'

Fastly

HTTP/3 is enabled per service via the Edge Cloud Platform dashboard or VCL configuration. Fastly auto-advertises Alt-Svc for HTTP/3 endpoints once enabled.

Vercel

HTTP/3 is enabled by default on Vercel's Edge Network for all projects. No configuration is required; clients that support QUIC will negotiate it automatically via Alt-Svc.

Self-hosted (Nginx, Caddy)

Nginx supports HTTP/3 from version 1.25 with the quic and http3 directives. Caddy enables HTTP/3 by default with zero configuration on any HTTPS site.

Caddyfile — HTTP/3 enabled automatically

example.com {

root * /var/www/html

file_server

}

When to use which

Use HTTP/2 (with HTTP/1.1 fallback) when:

  • Your traffic is dominated by bots and crawlers (>40% of requests)
  • Your origin sits behind a corporate network that rate-limits UDP
  • You need predictable CPU usage per request at scale
  • Your CDN does not offer mature HTTP/3 support

Use HTTP/3 (with HTTP/2 fallback) when:

  • Most traffic is from human visitors on mobile networks
  • You want connection migration for mobile users switching between Wi-Fi and cellular
  • You serve small, latency-sensitive payloads (API responses, ad creatives)
  • Your CDN handles QUIC offload at the edge

Best practice for AI search optimization: Enable both. Serve HTTP/3 for visitors and let AI crawlers continue to use HTTP/2. Do not break HTTP/2 — it is the protocol AI bots actually use.

Common mistakes

  • Disabling HTTP/2 to force HTTP/3 adoption. Crawlers will fall back to HTTP/1.1, losing multiplexing entirely.
  • Blocking UDP/443 at the firewall. This silently breaks HTTP/3 for visitors while leaving HTTP/2 untouched. Audit firewall rules before enabling QUIC.
  • Treating HTTP/3 as an AI search ranking signal. No major AI platform documents transport version as a citation signal. Focus on response latency, freshness, and structured data instead.
  • Forgetting Alt-Svc cache TTL. A misconfigured Alt-Svc: h3=":443"; ma=86400 directs clients to a non-existent QUIC endpoint for 24 hours after rollback. Set short max-age during testing.

FAQ

Q: Do AI crawlers prefer HTTP/3?

No. As of 2026, the major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended) negotiate HTTP/2 by default and do not opt into HTTP/3 even when origins advertise Alt-Svc: h3. Cloudflare's multi-year analysis confirms search and AI bots largely ignore HTTP/3.

Q: Will serving HTTP/3 improve my AI search rankings?

There is no published evidence that any AI platform uses transport version as a citation or ranking signal. AI search visibility depends on content quality, structured data, freshness signals, and robots.txt configuration — not on whether you serve QUIC.

Q: Should I disable HTTP/2 to force HTTP/3?

No. Disabling HTTP/2 forces clients (including AI bots) to fall back to HTTP/1.1, which lacks multiplexing and is strictly worse for crawl efficiency. Always keep HTTP/2 enabled alongside HTTP/3.

Q: How do I verify which protocol a crawler used?

Check access logs for the protocol field. On Cloudflare, the cf.tlsVersion and cf.httpProtocol log fields record HTTP/2 or HTTP/3. On Nginx, use $server_protocol in the log format. On Fastly, the req.proto VCL variable returns the negotiated version.

Q: Does HTTP/3 affect crawl budget?

Not directly. Crawl budget is governed by server response time, error rate, and Crawl-Delay directives. If HTTP/3 reduces your origin response time for human visitors, freed-up server resources may indirectly help bot serving — but the protocol itself is not a budget signal.

: Cloudflare Radar 2025 Year in Review — verified 2026-05-03 — supports HTTP/3 vs HTTP/2 share data. https://blog.cloudflare.com/radar-2025-year-in-review/

: Cloudflare, "Examining HTTP/3 usage one year on" — verified 2026-05-03 — supports bot adoption pattern. https://blog.cloudflare.com/http3-usage-one-year-on/

: Cloudflare, "Comparing HTTP/3 vs HTTP/2 Performance" — verified 2026-05-03 — supports payload-size benchmark numbers. https://blog.cloudflare.com/http-3-vs-http-2/

: CloudPanel, "HTTP/3 vs HTTP/2 Performance" — verified 2026-05-03 — supports lossy-network benchmark. https://www.cloudpanel.io/blog/http3-vs-http2/

: DebugBear, "HTTP/3 vs HTTP/2 Performance: Is the Upgrade Worth It?" — verified 2026-05-03 — supports middlebox/UDP nuance. https://www.debugbear.com/blog/http3-vs-http2-performance

: Ian Duncan, "HTTP/3 Is Not Always Faster Than HTTP/2" — verified 2026-05-03 — supports userspace stack cost. https://iankduncan.com/engineering/2026-02-10-http3-not-always-faster/

: Cloudflare, "HTTP/3 (with QUIC)" docs — verified 2026-05-03 — supports CDN config steps. https://developers.cloudflare.com/speed/optimization/protocol/http3/

: dev.to, "HTTP/3 and QUIC in Production: A Practical Deployment Guide for 2026" — verified 2026-05-03 — supports Caddy zero-config. https://dev.to/linou518/http3-and-quic-in-production-a-practical-deployment-guide-for-2026-3n8e

Related Articles

specification

Accept-Encoding (Brotli, Gzip) for AI Crawlers

Specification for serving Brotli, gzip, and zstd to AI crawlers via Accept-Encoding negotiation: which bots support which codecs, fallback rules, and Vary handling.

reference

AI Crawler IP Allowlist Reference

Reference list of official AI crawler IP range endpoints, user agents, and reverse-DNS verification methods for GPTBot, ClaudeBot, PerplexityBot, Googlebot, and more.

comparison

Brotli vs Gzip Compression for AI Crawlers

Brotli vs Gzip for AI crawler responses: ratio, CPU cost, Accept-Encoding negotiation, AI bot support, and CDN configuration patterns.

Topics
Stay Updated

GEO & AI Search Insights

New articles, framework updates, and industry analysis. No spam, unsubscribe anytime.