Crawl-to-Refer Ratio: Measure What Each AI Engine Takes and Gives Back

Last checked: October 27, 2026.

Open your server logs in 2026 and AI bots are everywhere. GPTBot reading the docs, ClaudeBot in the archives, PerplexityBot fetching last week's post. The reflexive question is "should I block them?" and the honest answer is: you cannot decide without a number.

That number exists. Cloudflare introduced the crawl-to-refer ratio in July 2025 to describe what it saw across its network. This page explains how to compute it for your own site, engine by engine, and what it can and cannot tell you.

Definition

Crawl-to-refer ratio = AI crawler requests received ÷ human visits referred back by that engine, over the same period.

It measures the bargain each AI engine offers you. A ratio of 50:1 means an engine made 50 crawl requests for every visitor it sent you. The lower the ratio, the better the deal. An engine that crawls heavily and refers nothing is extracting; an engine that crawls lightly and refers steadily is distributing.

The ratio only makes sense per engine, because the deal varies wildly between them, and that variance is exactly the decision-making information you need.

How to calculate it

You need two numbers per engine, same time window (a month is right):

1. Crawl requests : from server or CDN logs, count requests by user agent family. Group them by company: OpenAI = GPTBot + OAI-SearchBot + ChatGPT-User; Anthropic = ClaudeBot + Claude-SearchBot + Claude-User; and so on (the full user agent table is here ↗).

2. Referred visits : from analytics, count sessions by AI referrer (chatgpt.com, perplexity.ai, claude.ai...) per engine ↗.

Illustrative example, one month:
OpenAI crawls: 14,200 requests  | ChatGPT referrals: 310 visits → ratio 46:1
Perplexity crawls: 1,900        | Perplexity referrals: 95     → ratio 20:1
ByteDance crawls: 22,000        | referrals: 0                 → ratio ∞ (pure extraction)

Two honest caveats. First, referrer-based counts undercount real influence (app clicks and post-answer brand searches carry no AI fingerprint), so your true ratios are better than measured. Second, training crawlers pay you back on a delay measured in model versions, not weeks. The ratio is a flashlight, not a courtroom.

The trap in the numerator: user agents are not evidence

A common error in the top half makes every engine look worse than it is.

A user agent is a claim, not an identity. curl -A "GPTBot" https://yoursite.com takes five seconds, and scrapers do it constantly, sometimes to look legitimate, sometimes so that the block they earn lands on OpenAI's name instead of theirs. Grep your logs for GPTBot and you are counting OpenAI plus an unknown number of impostors. Your crawl count is inflated, so your ratio is pessimistic, and you may be about to block an engine over crawling it never did.

The verification is at the network layer, and there is no shortcut:

• Published IP ranges for the engines that publish them (OpenAI, Anthropic, Perplexity). A GPTBot request from outside OpenAI's range is a forgery.
• Reverse DNS for the ones that do not (Google, Bing, Apple): resolve the requesting IP to a hostname, then resolve that hostname back to the same IP.

A hit that fails the check is not a crawl by that engine, and it does not belong in the numerator. Datalenk runs both checks on every bot hit and marks the failures as spoofed, so the crawl number you divide by is the number that actually happened. It is a dull detail that changes the answer.

What the public figures say, and why they disagree

The only regular public source is Cloudflare, measured across the sites on its network. In its July 2025 post, Cloudflare put Anthropic at about 70,900 HTML page requests for each referral in the week of June 19 to 26, 2025, and warned that the figure may overstate the imbalance because Claude's native apps send no referrer, "it is unclear by how much".

Those numbers travel badly. In September 2026 Search Engine Journal counted seven different figures quoted for Anthropic alone, from 2,237:1 to 70,900:1, the gap coming from different time windows, different groupings of bots and missing app referrals rather than from errors.

Two practical consequences. There is no accepted benchmark for a "good" ratio, so compare engines on your own site rather than against a number from someone else's network. And always state the window and the bots you grouped when you share yours.

What to do with your ratio

1. Good ratio (engine refers meaningfully) : let everything in, and invest in citable content for that engine; it is buying distribution with your bandwidth, cheaply.
2. Bad ratio on a search/user engine : usually a visibility problem on your side rather than malice: blocked search bots, client-rendered content the bots cannot read, or pages that are not quotable. Fix access first (check your robots.txt against the current bot list ↗ and confirm your key pages are server-rendered), content shape second.
3. Bad ratio on a training-only crawler : a values-and-bandwidth decision. Blocking GPTBot does not touch your ChatGPT search visibility, because search is handled by OAI-SearchBot.
4. Track the trend, not the snapshot. Engines change their referral behavior in steps, with each product release that adds or removes links in answers. A taker can become a giver in one release, which is why this belongs in a dashboard rather than a quarterly log dive.

Where the two halves come from (and what Datalenk actually gives you)

Let us be precise, because the whole point of this page is a number people get wrong.

Datalenk does not print a "crawl-to-refer ratio" figure on a card in the dashboard. What it does is put both halves of the fraction in the same place, on the same time window, per engine, which is the part that is genuinely hard:

• The numerator: verified crawls per engine. This is collected server-side, through a small beacon in your middleware or edge function, because a crawler never runs your analytics script (JavaScript-only tools are structurally blind here, no matter what they claim). Every hit is checked against published IP ranges or reverse DNS, and spoofs are flagged rather than counted.
• The denominator: referral visits per engine, the AI Assistant channel broken out by ChatGPT, Perplexity, Claude, Gemini and the rest. Plus, because visits are not the real prize, the revenue those visits produced.

The division is yours to do once the two columns are side by side, on the same window. For the revenue side of AI referrals, see ↗.

Get both halves of your ratio in one dashboard. Verified AI crawls per engine, referral visits per engine, and the revenue behind them. 14-day trial, card required. Start your trial.

FAQ

What is a crawl-to-refer ratio? The number of requests an AI engine's crawlers make to your site divided by the human visits that engine refers back, over the same period, per engine. It measures what each AI engine takes versus what it gives.

What is a good crawl-to-refer ratio? There is no accepted benchmark. Figures published from Cloudflare's data for OpenAI and Anthropic run from over a thousand to tens of thousands of crawls per referral, depending on the engine and the week. Compare engines on your own site and watch the trend for each one.

Why is my ratio so bad? Four usual causes, and check the last one first: you are counting spoofed user agents as real crawls, which inflates the numerator (verify by IP, not by user agent string). Then the real ones: search bots blocked in robots.txt while training bots roam free, JavaScript-rendered content AI crawlers cannot read, or content that answers nothing quotable. Access problems first, content problems second.

Can I compute this from my analytics tool? Only if it sees bots, and a JavaScript snippet does not: crawlers never execute the script. You need a server-side source for the crawl half (raw logs, a CDN export, or a beacon in your middleware) and your analytics for the referral half, on the same window, per engine.

Does blocking high-ratio bots hurt my AI visibility? Blocking training bots (GPTBot, ClaudeBot, Bytespider, CCBot) does not affect AI search citations. Blocking search and user-fetch bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) absolutely does. Know which type you are blocking ↗.