AI Crawlers: What GPTBot, OAI-SearchBot and ClaudeBot Do

An AI crawler is a bot that an AI company such as OpenAI, Anthropic, Perplexity or Google runs to fetch web pages, whether to train its models, to supply its AI search answers, or to load a page when a user asks for it. Each goes by a user agent name, such as GPTBot or PerplexityBot, which is what a robots.txt rule addresses.

Purpose is what matters for visibility: search crawlers decide whether a site can appear in AI answers, and training crawlers do not. Their operators say both kinds honor robots.txt, while some user-triggered fetchers may not.

Four companies run the main AI crawlers, and each one documents its user agents and what they do.

Sources: OpenAI, Anthropic, Perplexity and Google crawler documentation, checked September 2026.

The most important distinction is between AI search crawlers, which feed live answers, and training crawlers, which collect training data. OpenAI states that GPTBot is for training and that blocking it does not affect ChatGPT search, which relies on OAI-SearchBot. Google says Google-Extended controls whether content is used to train Gemini models and does not affect inclusion or ranking in Google Search. Perplexity says PerplexityBot is for search and is not used to train foundation models.

So a company can block every training crawler and still appear in AI search answers, as long as the search crawlers are allowed. The reverse mistake is more common: a blanket block on "AI bots" that takes out OAI-SearchBot or PerplexityBot along with the training crawlers, and removes the site from those answers.

As a starting point, this robots.txt example allows the AI search crawlers and blocks the training crawlers. Swap the training rules to Allow if you are comfortable with your content being used for training. Googlebot is covered by the catch-all rule at the end.

# Search and answer crawlers: allow
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training crawlers: your decision (shown blocked here)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Allow: /

Two checks after any change: confirm the file returns cleanly at /robots.txt, and confirm your CDN or firewall is not blocking the same bots at the network level. Bot protection that rejects unfamiliar crawlers is one of the most common reasons a site never appears in ChatGPT or Perplexity answers.

Allowing AI crawlers does not by itself get you cited in AI answers, because access makes you eligible, not visible. Being cited still depends on whether your pages answer the questions being asked, and on what other sites say about you. Our guides to AI citations and getting cited by ChatGPT cover what comes next. And an llms.txt file is not a substitute for crawler access; see llms.txt.

Crawler audits are part of the technical foundation in our AI search optimization engagements, because a single misconfigured rule can hide a site from an entire AI surface. The same audit runs in our B2B SaaS SEO engagements, where pricing, docs and comparison pages need to stay open to AI search crawlers.

These AI crawler FAQs cover what site owners ask before they change a robots.txt rule.

Should I block GPTBot?

The decision to block GPTBot depends on whether you want your content used to train OpenAI's models. Blocking GPTBot does not remove you from ChatGPT search, which uses OAI-SearchBot. Many companies allow the search crawlers and make a separate decision about training crawlers.

Does blocking Google-Extended remove me from AI Overviews?

No, blocking Google-Extended does not remove you from AI Overviews. Google says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search. To leave them, use the Search generative AI setting Search Console added for all sites on August 31, 2026, or snippet controls.

How do I check if AI crawlers can reach my site?

To see whether AI crawlers can reach your site, read your robots.txt for rules naming their user agents, then check your CDN or firewall bot settings, which can block crawlers even when robots.txt allows them. Server logs will show whether the bots are actually arriving.

Do AI crawlers read llms.txt?

AI crawlers rarely read llms.txt. In one 12-week server-log study, OpenAI's crawler requested robots.txt 3,990 times and llms.txt 7 times. See our llms.txt guide for the details.

Why do user-triggered fetchers ignore robots.txt?

Some companies treat a fetch requested by a person, such as pasting a link into a chat, as a user action rather than crawling, which is what robots.txt covers. OpenAI and Perplexity say their user-triggered fetchers may not follow robots.txt; Anthropic says all of its bots do.

• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity: Perplexity crawlers
• Google Search Central: Google common crawlers (Google-Extended)
• EZY Research: llms.txt and robots.txt requests by AI crawlers

To see if your brand shows up in the answers AI crawlers feed, start with the Category Salience Brief, the first step in every High Salience engagement: your commercial questions run across Google and the AI surfaces, competitors side by side, and a ranked list of what to fix first.