AI Crawlers in 2026: GPTBot, ClaudeBot, PerplexityBot and the Others

Table checked against each company's crawler documentation on October 27, 2026. The sources are listed at the end.

Every day, your website gets visited by bots you never invited. Some of them are building the next GPT. Some are fetching your pages so an AI assistant can quote you in a conversation happening right now. Some take your content and send nothing back.

This guide covers the crawlers run by the main AI companies, what each one does according to its operator, and how to allow or block it.

If you only remember one thing from this page, make it this: AI crawlers are not one category. They are three. Treating them as one is how sites end up blocking the bots that send them visitors while letting through the ones that only train.

The three types of AI crawlers

OpenAI and Anthropic each document three separate bots, and Perplexity two, following the same split.

1. Training crawlers collect content to train future models. Blocking them keeps your future content out of training datasets, but it has no effect on data that was already collected. Examples: GPTBot, ClaudeBot, Bytespider.

2. Search index crawlers build the retrieval indexes behind AI search experiences. If you block these, your site stops appearing (and being cited) in AI search answers. Examples: OAI-SearchBot, Claude-SearchBot.

3. User-fetch agents retrieve a page on demand, because a human just asked the assistant something and the assistant needs your page to answer. These visits can turn into real referral traffic, since the assistant links to you as a source. Examples: ChatGPT-User, Claude-User, Perplexity-User.

The strategic consequence is simple: most sites benefit from allowing types 2 and 3 (visibility, citations, referrals) while making a deliberate choice about type 1 (training). Blocking everything because "AI is stealing content" also removes you from the answers people now read instead of a results page.

The reference table

User agent	Company	Type	What it does	robots.txt
GPTBot	OpenAI	Training	Collects content for model training	Respected
OAI-SearchBot	OpenAI	Search index	Indexes pages for ChatGPT search results	Respected
ChatGPT-User	OpenAI	User-fetch	Fetches pages live during ChatGPT conversations	Not guaranteed (OpenAI: "Because these actions are initiated by a user, robots.txt rules may not apply")
OAI-AdsBot	OpenAI	Ads review	Visits landing pages submitted as ChatGPT ads to check them; OpenAI says its data is not used for training	Respected
ClaudeBot	Anthropic	Training	Collects content for Claude model training	Respected
Claude-SearchBot	Anthropic	Search index	Indexes pages for Claude's search features	Respected
Claude-User	Anthropic	User-fetch	Fetches pages when Claude users request them	Respected
PerplexityBot	Perplexity	Search index	Indexes pages for cited answers	Respected
Perplexity-User	Perplexity	User-fetch	Fetches pages on user request	No (Perplexity: "this fetcher generally ignores robots.txt rules")
Google-Extended	Google	Training control	Not a separate bot: a robots.txt token that tells Google not to use your content for Gemini training. Crawling still happens via Googlebot	Respected
GoogleOther	Google	Misc/R&D	Generic crawler for Google research and product development	Respected
Applebot-Extended	Apple	Training control	Like Google-Extended: a token controlling Apple Intelligence training use, crawling happens via Applebot	Respected
Bytespider	ByteDance	Training	Collects content for ByteDance models	We found no crawler documentation from ByteDance; block at the server if it matters
CCBot	Common Crawl	Training (indirect)	Builds the open Common Crawl dataset used by many AI labs	Respected
Amazonbot	Amazon	Training + products	Used to improve Amazon products and services, which can include AI model development	Respected
Meta-ExternalAgent	Meta	Training	Collects content for Meta AI models	Respected
Meta-ExternalFetcher	Meta	User-fetch	Fetches individual links at a user's request for Meta AI features	Not guaranteed (Meta: it "may bypass robots.txt rules")
DuckAssistBot	DuckDuckGo	Answers	Fetches pages for DuckDuckGo's AI-assisted answers, which cite their sources; DuckDuckGo says the data is not used for training	Respected

A note on the two "-Extended" entries: webmasters often look for a "Google-Extended bot" in their logs and never find it. That is expected. Google-Extended and Applebot-Extended are opt-out switches, not crawlers. The crawling is done by Googlebot and Applebot, which you almost certainly do not want to block.

The caveat that invalidates most crawler reports: user agents are just a claim

Everything in that table is a string a client chooses to send. A user agent is declarative, not proof. Anyone can send GPTBot from a laptop in five seconds with curl -A, and scrapers routinely do exactly that, sometimes to look legitimate, sometimes to get themselves blocked on purpose so the block lands on OpenAI's reputation rather than theirs.

The consequence is uncomfortable if you have ever counted bots in a log file: a raw user agent count overstates how much the real engines crawl you. Part of what you attribute to GPTBot is not OpenAI at all.

The only thing that settles it is the network layer:

• Published IP ranges. OpenAI, Anthropic and Perplexity publish the address ranges their crawlers use, as JSON files linked from their crawler pages. A request claiming to be GPTBot from outside OpenAI's range is a forgery, full stop.
• Reverse DNS. Google, Bing and Apple do not publish fixed ranges, so the check is a reverse DNS lookup on the requesting IP followed by a forward lookup back to the same address.

Anything that fails those checks is a spoof, and it should be labeled as one rather than quietly inflating your numbers. That verification is server-side work by definition, which is also why analytics tools that live in a JavaScript snippet cannot do it: crawlers do not run JavaScript, so they never reach the script in the first place. Datalenk does the check on the server, keeps verified engine hits apart from spoofed ones, and sorts what remains into the same three types this article opens with (training, search index, user-fetch), automatically, per engine.

OpenAI's crawlers, explained

OpenAI documents the clearest separation between its bots, and ChatGPT is the assistant most site owners ask about first.

• GPTBot is the training crawler. Blocking it is a values decision more than a traffic decision: it does not remove you from ChatGPT search and does not stop citations. It only keeps future training runs from ingesting your content.
• OAI-SearchBot is what gets you into ChatGPT's search index. If you care about being cited and clicked from ChatGPT answers, this bot must be allowed. Blocking it while allowing GPTBot is the exact opposite of what most site owners actually want.
• ChatGPT-User shows up when a real person's conversation triggered a fetch of your page. Think of it as a click that has not happened yet: the assistant reads your page, summarizes it, and links you as a source. OpenAI states that robots.txt rules may not apply to it, because a user started the action.
• OAI-AdsBot only visits landing pages submitted as ads in ChatGPT, to review them.

Anthropic's crawlers, explained

Anthropic mirrors the same pattern: ClaudeBot for training, Claude-SearchBot for search indexing, Claude-User for live fetches during Claude conversations. Anthropic says all of them honor robots.txt, supports the non-standard Crawl-delay directive, and publishes its crawler IP addresses. You may also still see legacy tokens like Claude-Web or anthropic-ai in older logs and robots.txt templates; the three bots above are the current canonical set.

Perplexity's crawlers, explained

Perplexity is an answer engine first, so its crawling exists almost entirely to cite sources. PerplexityBot builds the index, Perplexity-User fetches on demand. Worth knowing: Perplexity's own documentation says Perplexity-User "generally ignores robots.txt rules" because a user requested the fetch, and in August 2025 Cloudflare reported undeclared Perplexity crawlers reaching sites that had blocked its declared bots, which Perplexity disputed. If you need a hard guarantee against Perplexity fetches, a robots.txt line is not enough; you will need server-level blocking (user agent filtering or a bot management layer).

Should you block AI crawlers?

There is no universal answer, but there is a rational framework: compare what each bot takes (crawl load, content value) with what it gives back (citations, referral traffic).

That ratio varies enormously by bot type. Search and user-fetch bots send traffic back; training bots do not. This is exactly what the crawl-to-refer ratio measures ↗: for each engine, what it fetches from you against the visitors it sends back.

Practical guidance by site type:

• Publishers and content businesses: consider blocking training bots (your content is your asset) while allowing search and user-fetch bots (citations are distribution).
• SaaS, e-commerce and service businesses: allow everything except the chronically non-compliant. Being recommended by ChatGPT when someone asks "what tool should I use for X" works like word of mouth, and you want your docs and comparison pages in every index.
• Sites with paywalled or licensed content: block training and consider licensing deals; allow search bots only for free content sections.

Copy-paste robots.txt templates

Template 1: maximize AI visibility, opt out of training

# Allow AI search and user-fetch bots (citations and referrals)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

Template 2: allow everything (maximum AI distribution)

No robots.txt entries needed. Every compliant bot crawls by default. Consider adding an llms.txt file instead, to guide what AI systems should read first.

Template 3: block everything AI

Take Template 1, set every entry to Disallow: /, and remember two caveats: already-trained models keep what they learned, and non-compliant bots ignore the file anyway. For real enforcement, filter user agents at the server or CDN level.

How to see which AI bots crawl YOUR site

Everything above describes the general picture. Your site has its own reality, and it is probably different from what you expect.

Three ways to find out, from quick to thorough:

1. Check your robots.txt against the current bot list (the table above changes; your file probably has not).
2. Grep your server logs for the user agent tokens in the reference table and count requests per bot per week. Remember the previous section: this is an upper bound, because it counts spoofs as real crawls.
3. Measure it on the server, continuously. A crawler never executes your analytics script, so no JavaScript-only tool will ever show you one, ours included. The fix is a small beacon in your middleware or edge function that posts the request's user agent and IP to your analytics: Datalenk gives you the snippet, verifies each hit against published IP ranges and reverse DNS, flags the forgeries, and puts verified crawls per engine next to the referral visits per engine and the revenue those visits produced ↗. Same place, same time window, so the comparison is finally possible.

FAQ

What is GPTBot? GPTBot is OpenAI's training crawler. It collects web content that may be used to train future models. It respects robots.txt.

Does blocking GPTBot remove my site from ChatGPT? No. ChatGPT search visibility is controlled by OAI-SearchBot, and live page fetches are done by ChatGPT-User. Blocking GPTBot only opts your future content out of model training.

What is the difference between GPTBot and ChatGPT-User? GPTBot crawls proactively to gather training data. ChatGPT-User fetches a specific page on demand because a user's conversation needs it, which is the visit type that produces citations and referral clicks.

Do AI crawlers respect robots.txt? The training and search crawlers of the major companies say they do (OpenAI, Anthropic, Google, Apple, Amazon, Meta, Common Crawl). Several user-triggered fetchers say they may not: ChatGPT-User, Perplexity-User and Meta-ExternalFetcher. For guarantees, block at the server level.

Can AI crawler user agents be faked? Yes, trivially. The user agent is a string the client chooses to send, so any scraper can claim to be GPTBot. The only real verification is at the IP level: published crawler IP ranges for OpenAI, Anthropic and Perplexity, reverse DNS for Google, Bing and Apple. Counts based on user agent alone overstate how much the real engines crawl you.

Should I use llms.txt or robots.txt for AI bots? Both, for different jobs. robots.txt controls access (allow/block). llms.txt is an emerging standard that tells AI systems what your site is about and which pages matter most, improving how you are represented in AI answers.

Sources

Each company's own crawler documentation, checked on October 27, 2026: OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, Common Crawl, DuckDuckGo. Perplexity's undeclared crawlers: Cloudflare, August 4, 2025.