AI Crawlers Explained: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended

Sources: OpenAI, Anthropic, Perplexity and Google crawler documentation, checked September 2026.

The most important distinction is between crawlers that feed live answers and crawlers that collect training data. OpenAI states that GPTBot is for training and that blocking it does not affect ChatGPT search, which relies on OAI-SearchBot. Google says Google-Extended controls whether content is used to train Gemini models and does not affect inclusion or ranking in Google Search. Perplexity says PerplexityBot is for search and is not used to train foundation models.

So a company can block every training crawler and still appear in AI search answers, as long as the search crawlers are allowed. The reverse mistake is more common: a blanket block on "AI bots" that takes out OAI-SearchBot or PerplexityBot along with the training crawlers, and removes the site from those answers.

This example allows the search crawlers and blocks the training crawlers. Swap the training rules to Allow if you are comfortable with your content being used for training. Googlebot is covered by the catch-all rule at the end.

# Search and answer crawlers: allow
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training crawlers: your decision (shown blocked here)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Allow: /

Two checks after any change: confirm the file returns cleanly at /robots.txt, and confirm your CDN or firewall is not blocking the same bots at the network level. Bot protection that rejects unfamiliar crawlers is one of the most common reasons a site never appears in ChatGPT or Perplexity answers.

Allowing a crawler makes you eligible, not visible. Being cited still depends on whether your pages answer the questions being asked, and on what other sites say about you. Our guides to AI citations and getting cited by ChatGPT cover what comes next. And an llms.txt file is not a substitute for crawler access; see llms.txt.

Crawler audits are part of the technical foundation in our AI SEO engagements, because a single misconfigured rule can hide a site from an entire AI surface.

Should I block GPTBot?

It depends on whether you want your content used to train OpenAI's models. Blocking GPTBot does not remove you from ChatGPT search, which uses OAI-SearchBot. Many companies allow the search crawlers and make a separate decision about training crawlers.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search and are controlled through Googlebot and snippet settings.

How do I check if AI crawlers can reach my site?

Read your robots.txt for rules naming these user agents, then check your CDN or firewall bot settings, which can block crawlers even when robots.txt allows them. Server logs will show whether the bots are actually arriving.

Do AI crawlers read llms.txt?

Rarely. In one 12-week server-log study, OpenAI's crawler requested robots.txt 3,990 times and llms.txt 7 times. See our llms.txt guide for the details.

Why do user-triggered fetchers ignore robots.txt?

Some companies treat a fetch requested by a person, such as pasting a link into a chat, as a user action rather than crawling. OpenAI and Perplexity say their user-triggered fetchers may not follow robots.txt; Anthropic says all of its bots do.

• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity: Perplexity crawlers
• Google Search Central: Google common crawlers (Google-Extended)
• EZY Research: llms.txt and robots.txt requests by AI crawlers

Every High Salience engagement starts with the Category Salience Brief: your commercial questions run across Google and the AI surfaces, competitors side by side, and a ranked list of what to fix first.