AI Crawler Cost Attribution Framework: Allocating Compute and Bandwidth Across LLM Bots

TL;DR

LLM crawlers consume real CPU, bandwidth, and edge-request budget, but most teams still treat them as one undifferentiated category. The AI Crawler Cost Attribution Framework instruments traffic per user-agent, derives a $/1k requests and $/GB figure for each bot, compares that cost against citation and referral benefit, and routes each bot into one of four policy tiers: Allow, Throttle, Charge, or Block. The output is a per-bot ledger your infra and finance teams can defend.

Why per-bot attribution matters

A single "AI bots" line item hides three different economic realities. User-facing fetchers like ChatGPT-User and PerplexityBot retrieve content because a human asked a question, so each request maps to a potential referral. Training crawlers like GPTBot, ClaudeBot, and Google-Extended bulk-scrape for model training and rarely send traffic back. Aggressive scrapers ignore robots.txt entirely and behave more like an unintentional DDoS.

Several signals make this attribution urgent in 2026:

• AI crawlers and scrapers were linked to an 86% increase in general invalid traffic year over year, raising raw infrastructure load even when conversions are flat.
• A Wharton and Rutgers working paper found that publishers who blocked LLM crawlers via robots.txt lost roughly 7% of weekly human traffic within six weeks, suggesting blanket blocks have measurable opportunity cost.
• Cloudflare's pay-per-crawl program turned crawl access into a priced primitive, with HTTP 402 Payment Required responses and a single domain-wide price per request.

Without attribution, you cannot tell whether a $4,000 monthly bandwidth bill is paying for citations on Perplexity or subsidizing Bytespider. The framework below makes that ledger explicit.

The framework

The framework has four steps. Treat each step as a pipeline stage that produces an artifact the next stage consumes.

Step 1 — Instrument per-bot telemetry

You cannot attribute what you cannot measure. Capture, at the edge, for every request:

• Verified user-agent (cross-checked against the bot operator's published IP ranges or signed bot tokens)
• Bytes served (response body + headers)
• Origin CPU time or worker compute units consumed
• Cache status (hit, miss, revalidated)
• HTTP status (only 2xx counts as a billable, value-bearing fetch)

Normalize user-agents into a stable bot identity table. Variants like GPTBot/1.2, OAI-SearchBot, and ChatGPT-User are different economic actors and must not be collapsed.

Step 2 — Attribute unit cost per bot

For each bot b over a window (weekly is a good default), compute:

cost(b) = bandwidth_GB(b) * $/GB

+ origin_requests(b) * $/origin_request

+ edge_requests(b) * $/edge_request

+ cpu_seconds(b) * $/cpu_second

Derive $/GB, $/origin_request, $/edge_request, and $/cpu_second from your actual cloud and CDN invoices, not list prices. Then express two unit metrics that travel well across teams:

• Cost per 1k requests — comparable across bots regardless of volume
• Cost per GB served — exposes bots that pull large media or full HTML repeatedly without revalidation

Keep cache-hit traffic in a separate column. A bot whose traffic is 95% cached at the edge costs an order of magnitude less than one forcing origin fetches.

Step 3 — Estimate benefit per bot

Benefit is harder than cost, but you only need a defensible proxy. Combine three signals:

1. Citation rate. Track citations of your domain in the surface tied to that bot (ChatGPT answers, Claude responses, Perplexity citations, Google AI Overviews). Sample weekly with a fixed prompt set.
2. Referral traffic. Count human sessions whose referrer or first-touch attribution maps to the bot's parent platform.
3. Crawl-to-refer ratio. Pages crawled by the bot divided by referrals it sent back. A 100:1 ratio means 100 fetches per single human visit.

Convert to a benefit score:

benefit(b) = referrals(b) * $/referral

+ citations(b) * $/citation

$/referral is straightforward — pull it from your existing organic-traffic LTV model. $/citation is fuzzier; a defensible starting point is the cost of an equivalent paid placement on the same surface, discounted for clickthrough.

Step 4 — Map each bot to a policy tier

With cost(b) and benefit(b), classify each bot into one of four tiers:

• Allow — benefit(b) >> cost(b). Typical for user-facing fetchers (ChatGPT-User, PerplexityBot, Claude-Web, Google-Extended for AI Overviews).
• Throttle — benefit positive but volume disproportionate. Apply rate limits or cache-only responses (e.g. 100 req/min for GPTBot, 200 req/min for Google-Extended).
• Charge — bulk training crawler with low referral but real licensing value. Cloudflare's 402 flow with a flat per-request price is the operational primitive; Stack Overflow's pay-per-crawl rollout is a public reference for this tier.
• Block — cost(b) > 0 and benefit(b) ≈ 0, especially for crawlers that ignore robots.txt (e.g. Bytespider and unverified scrapers).

Record each decision with the cost and benefit numbers that justified it, and re-run the calculation on a 30-day cadence.

Worked example

A mid-size documentation site over a 30-day window:

The Allow tier carries the citation upside; the Charge tier converts otherwise sunk training cost into licensing revenue via pay-per-crawl; the Block tier removes pure cost. Without attribution, all five would have shared one undifferentiated bandwidth budget.

Operating the framework

• Re-tier on a 30-day cadence. Bot behavior changes when vendors ship new model releases or new search products.
• Pin verification logic. Any bot that fails IP or signature verification automatically falls into Block, regardless of declared user-agent.
• Watch the cache layer. Edge caching is the single biggest lever on cost(b) and lets you keep Allow tiers wide.
• Tie the ledger to citation-readiness work. A bot you allow only pays back if your pages are structured to be cited; pair this framework with a citation-readiness checklist.
• Coordinate with finance. Export the per-bot ledger monthly so infra spend is reconciled against AI-attributable revenue.

Common pitfalls

• Collapsing user-agents. Treating GPTBot and ChatGPT-User as one bot destroys the cost-vs-benefit signal.
• Counting cached and origin traffic the same. Cache hits at the edge are nearly free; mixing them inflates Allow-tier costs and triggers unnecessary blocks.
• Blanket blocks. Robots.txt blocks of all LLM bots correlate with measurable human-traffic loss and rarely protect content from determined scrapers.
• Single-price thinking on pay-per-crawl. Cloudflare's current implementation uses one domain-wide price across all charged crawlers; build your tier policy around that constraint rather than imagined per-bot price ladders.

FAQ

Q: What is AI crawler cost attribution?

AI crawler cost attribution is the practice of assigning a measurable infrastructure cost — bandwidth, origin compute, and edge requests — to each individual LLM bot that hits your site, then comparing that cost to the referrals and citations the bot generates so you can decide whether to allow, throttle, charge, or block it.

Q: Which AI crawlers should I never block?

User-facing fetchers that retrieve a page only when a human asks a question — ChatGPT-User, Claude-Web, PerplexityBot, OAI-SearchBot, and Google-Extended for AI Overviews — should default to Allow, because each request is tied to a likely referral or citation event.

Q: How does Cloudflare pay-per-crawl fit into this framework?

Pay-per-crawl is the operational primitive for the Charge tier. You set a single domain-wide per-request price, mark training crawlers as Charge, and Cloudflare returns 402 Payment Required with pricing to bots that have not paid; charges apply only to successful 2xx responses.

Q: What is a healthy crawl-to-refer ratio?

There is no universal target, but lower is better. A ratio in the low hundreds for training crawlers is normal because they index broadly. For user-facing bots, ratios above a few hundred to one suggest you are subsidizing scraping without earning referrals, and the bot should drop into Throttle or Charge.

Q: How often should I recompute the ledger?

Monthly is the practical default. Recompute sooner when a major model vendor ships a new product (which often introduces or renames a crawler), when you change CDN or origin pricing, or when a new bot exceeds 1% of total request volume.

: Wharton and Rutgers working paper on the impact of robots.txt blocking on news publisher traffic, 2026.

: Cloudflare, Introducing pay per crawl, blog and developer documentation, 2025-2026.

: SEOmator, Crawl-to-refer ratio, GEO Data Report 2026.