2 articles
referenceadvanced5 min read
Conditional GET and ETag Handling for AI Crawlers
Conditional GET and ETag handling for AI crawlers: ETag generation, If-None-Match, If-Modified-Since, 304 Not Modified, and bandwidth-saving patterns.
referenceintermediate6 min read
HTTP Status Code Reference for AI Crawlers
HTTP status code reference for AI crawlers: how 2xx, 3xx, 4xx, 5xx codes affect GPTBot, ClaudeBot, PerplexityBot, and Googlebot indexing.