Retry-After and Rate Limit Headers for AI Crawlers

When GPTBot, ClaudeBot, or PerplexityBot exceeds your per-host quota, return 429 Too Many Requests with a Retry-After header and pair it with RateLimit / RateLimit-Policy so the crawler knows when and how often to come back; silent blocks and bare 5xx responses cause crawlers to deprioritize the host and drop.

TL;DR

Use 429 for quota breaches, 503 for maintenance windows, always include Retry-After. Advertise the policy via RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset (or the newer combined RateLimit / RateLimit-Policy per the IETF draft). Per-bot quotas of one request per 1-3 seconds are a sensible starting point. Verified bots that honor these signals stay in good standing.

Scope

This specification documents the response headers used to throttle AI crawler traffic without dropping citation eligibility. It applies to origin servers, CDN edge configurations, and WAF rules. Audience: SREs, platform engineers, and edge-network owners.

It does not cover robots.txt crawl-delay or pay-per-crawl economics; pair this spec with those controls when shipping a complete strategy.

Header semantics

Retry-After

Defined by MDN, Retry-After indicates how long the client should wait before retrying. Two formats are valid:

Delta-seconds: a non-negative integer (Retry-After: 120).
HTTP-date: an absolute RFC 7231 date (Retry-After: Sat, 03 May 2026 14:30:00 GMT).

Use delta-seconds for short, dynamic backoffs and HTTP-date for known maintenance windows. The header is meaningful with status codes 429, 503, and 301.

RateLimit-Limit / RateLimit-Remaining / RateLimit-Reset

These de facto headers, derived from draft-ietf-httpapi-ratelimit-headers, advertise the current quota:

RateLimit-Limit: 1000 — maximum requests per window.
RateLimit-Remaining: 423 — requests remaining in the current window.
RateLimit-Reset: 60 — seconds until the window resets (or HTTP-date).

The IETF draft has expired without RFC publication. Therefore, the three-header form remains widely deployed as a convention even though the working group moved toward a combined RateLimit plus RateLimit-Policy design (http.dev, 2025).

Combined RateLimit + RateLimit-Policy

The newer draft uses Structured Fields:

RateLimit-Policy: "default";q=1000;w=60
RateLimit: "default";r=423;t=42

Where q is the quota, w is the window in seconds, r is remaining, and t is time to reset. Adopt this when targeting standards-aware clients.

Status code patterns

Status	Meaning for AI crawlers	Required headers
429 Too Many Requests	Crawler exceeded per-host or per-IP quota.	Retry-After, `RateLimit-
503 Service Unavailable	Origin overloaded or in maintenance.	Retry-After
5xx without Retry-After	Crawlers retry with their own backoff schedule; repeated misses cause deprioritization.	None (avoid)
403 Forbidden	Permanent denial. Crawler will stop attempting.	None (use only for hard blocks)
404 Not Found	Resource gone; crawler removes URL from index.	None

Returning silent timeouts or connection resets is the worst outcome — the crawler treats the host as unstable, which can affect citation eligibility for the entire domain.

Per-bot quota strategy

Public AI crawlers vary in scrape intensity. ClaudeBot has been observed making nearly a million requests to single hosts in a day per public reports (InMotion Hosting, 2025). A defensible policy:

Default: 1 request per 1-3 seconds per User-Agent per hostname.
Aggressive crawlers (ClaudeBot, Amazonbot, Bytespider): tighten to 1 request per 5-10 seconds.
Real-time retrieval bots (OAI-SearchBot, ChatGPT user-triggered): allow burstier traffic with a per-minute quota of 60-120 requests so on-demand citations succeed.
Verified Googlebot and Bingbot: keep above all AI bots; they are the most rate-sensitive for SEO.

Match by full User-Agent string and reverse-DNS or published IP range, not partial substring, to avoid spoof-friendly throttling.

Edge configuration recipes

Cloudflare

Use the Rate Limiting Rules product. Match on (http.user_agent contains "GPTBot") and set 30 requests per 60 seconds. Action: Block with custom response code 429 and a custom response body. Cloudflare automatically appends Retry-After and X-RateLimit- style headers.

Fastly

Use VCL with the Edge Rate Limiting add-on or write custom logic in vcl_recv. Set Retry-After via a synthetic 429 response when the per-IP counter exceeds quota.

Nginx

limit_req_zone $http_user_agent zone=ai_bots:10m rate=20r/m;

server {

if ($http_user_agent ~ "GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot") {

set $is_ai_bot 1;

}

location / {

limit_req zone=ai_bots burst=10 nodelay;

limit_req_status 429;

add_header Retry-After 60 always;

add_header RateLimit-Limit 20 always;

}

Nginx returns 429 on quota breach; the Retry-After and RateLimit-Limit headers ride along on the synthetic response.

ModSecurity

The pattern published by InMotion Hosting targets specific User-Agents with one request per 3 seconds. Pair the rule with a setRespHeader action to attach Retry-After.

Monitoring crawler retry behavior

Log every 429 and 503 response with bot User-Agent, IP, and Retry-After value sent.
Track time-to-next-request per bot; well-behaved crawlers re-request after roughly the advertised interval.
Alert if a bot ignores Retry-After for more than three consecutive cycles — either the bot is misbehaving or your quota is misconfigured.
Compare citation share before and after policy changes (Bing Webmaster Tools, GSC, third-party AI visibility tools) to confirm throttling did not silently remove URLs.

Common pitfalls

Returning 429 without Retry-After — crawlers fall back to vendor-specific defaults, often longer than you intend.
Mixing delta-seconds and HTTP-date in the same response — invalid; pick one format.
Setting Retry-After: 0 — effectively meaningless; the crawler retries immediately and re-triggers the rate limit.
Throttling via connection: close instead of 429 — looks like an outage, not a quota signal.
Per-IP quotas for crawlers that rotate IPs — use User-Agent + reverse-DNS verification instead.
Differential throttling that hides the policy — if you serve different RateLimit-Limit values per bot, advertise that explicitly.

FAQ

Q: Should I always include Retry-After on a 429?

Yes. Per MDN, Retry-After is recommended on 429 and 503. Without it, well-behaved crawlers fall back to internal defaults that may be longer than necessary, reducing how often your content gets re-crawled and cited.

Q: What's the difference between RateLimit-* headers and Retry-After?

Retry-After answers "when should I retry _this_ request?" The RateLimit- headers answer "what is my overall quota and how much do I have left?" Use both: Retry-After for the immediate response, RateLimit- for ongoing client guidance.

Q: Will throttling AI crawlers hurt my citation visibility?

Not when done politely. Crawlers expect throttling and adjust their schedules. The damaging signal is silent failure (timeouts, resets, hard blocks without 429), which marks the host as unstable.

Q: How aggressive should my default rate limit be?

Start with 1 request per 1-3 seconds per AI bot. Tighten only if you observe sustained bandwidth pressure. ChatGPT user-triggered fetches and OAI-SearchBot deserve looser limits because they correspond to real user citations.

Q: Does Retry-After accept fractional seconds?

No. The header value is an integer number of seconds or an HTTP-date. Use 1 instead of 0.5.

Q: Can I use 503 instead of 429 for rate limiting?

429 is the correct status for quota-based throttling. 503` is reserved for service unavailability (overload, maintenance). Mixing them confuses crawler heuristics.