How to Allow or Block AI Crawlers in robots.txt

AI companies run several kinds of crawlers, and they don't all do the same thing. Some collect pages to train models. Some build an index that an AI search product uses to answer and cite. Some only fetch a page when a user asks the assistant to look at it. Blocking "AI bots" as one group can remove you from AI search results you actually want to appear in. This guide explains the categories, lists the main user agents as their operators document them, and gives robots.txt rules you can adapt.

Three kinds of AI crawler#

1. Training crawlers collect content that may be used to train or improve AI models. Blocking them limits future training use of your content. It doesn't remove anything a model has already learned.
2. Search and indexing crawlers build the index an AI product uses to find and cite pages in its answers. Blocking them can keep your pages out of that product's answers.
3. User-triggered fetchers visit a page because a person asked the assistant to read or check it. Some operators say robots.txt may not apply to these, because a human initiated the request.

Operators also use control tokens that aren't separate crawlers at all. They're names you can use in robots.txt to control how content collected by an existing crawler is used.

The main AI user agents#

The descriptions below follow each operator's own documentation at the time of writing. Operators update these pages, so check them before you rely on the details.

OpenAI (OpenAI crawler documentation):

• GPTBot: collects content that may be used to train OpenAI's generative AI foundation models. OpenAI states that disallowing GPTBot indicates a site's content should not be used for that training.
• OAI-SearchBot: used to surface websites in ChatGPT's search features. OpenAI says sites that opt out won't be shown in ChatGPT search answers, though they can still appear as navigational links.
• ChatGPT-User: used for actions a user initiates in ChatGPT. OpenAI notes that because these actions are user-initiated, robots.txt rules may not apply.

Anthropic (Anthropic crawler documentation):

• ClaudeBot: collects web content to improve Anthropic's models. It respects Disallow rules and the non-standard Crawl-delay extension.
• Claude-SearchBot: crawls to improve search result quality. Blocking it prevents your content from being indexed for that purpose.
• Claude-User: fetches pages when a Claude user asks a question. Anthropic says it honors robots.txt, and blocking it prevents retrieval of your content in response to user queries.

Perplexity (Perplexity crawler documentation):

• PerplexityBot: designed to surface and link websites in Perplexity's search results. Perplexity says it isn't used to crawl content for AI foundation models.
• Perplexity-User: supports user actions. Perplexity says it generally ignores robots.txt, because a user requested the fetch.

Google (Google's common crawlers documentation):

• Google-Extended: a robots.txt product token, not a separate crawler. Crawling is done with Google's existing user agents. The token lets you control whether content Google crawls may be used to train future generations of Gemini models. Google states that Google-Extended does not affect a site's inclusion in Google Search and isn't used as a ranking signal.

Apple:

• Applebot-Extended: Apple documents it as a token that does not crawl pages. Disallowing it opts content crawled by Applebot out of use for training Apple's foundation models, and pages that disallow it can still appear in Apple's search results. Applebot itself powers features such as Spotlight, Siri and Safari.

Common Crawl:

• CCBot: the crawler behind Common Crawl's openly available web archive. Common Crawl documents how to block it with robots.txt, and warns that other crawlers sometimes falsely identify themselves as CCBot.

How robots.txt groups work#

One rule catches many people out. Under the Robots Exclusion Protocol (RFC 9309), a crawler follows the group that matches its user agent most specifically. It falls back to the User-agent: * group only when no specific group matches. So if you add a group for GPTBot, GPTBot ignores everything in your * group. Repeat any rules you still want it to follow.

Example configurations#

Block training, allow AI search and user fetches. This is a common middle ground for publishers who want to be cited but not used for training:

# Training and training-use tokens
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# AI search crawlers: allowed, but keep private areas out
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /account/
Disallow: /checkout/

# Everyone else
User-agent: *
Disallow: /account/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

Block all listed AI agents from part of the site only:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Disallow: /premium/
Disallow: /members/

Allow everything. An empty Disallow: or no AI-specific groups at all lets compliant AI crawlers follow your normal rules.

After any change, test specific URLs against specific user agents with the Robots.txt Checker. A misplaced group or a typo in a user agent name is easy to miss by eye. Our robots.txt guide covers the syntax in more depth.

Trade-offs to weigh#

Reasons to block training crawlers:

• You sell content, research or data that you don't want reproduced by models.
• Your content is licensed and you need to control reuse.
• You want to reduce crawl load from bots that don't send traffic back.

Reasons to allow search and fetch agents:

• AI assistants increasingly answer questions with citations and links. If their search crawlers can't read your pages, those products can't cite you. See how brands get cited by AI.
• Blocking user-triggered fetchers means an assistant can't read your page even when a user shares your link.

Limits of blocking:

• Blocking now doesn't remove content already collected.
• Robots.txt is a request, not enforcement. Google puts it plainly: robots.txt instructions "cannot enforce crawler behavior," and while respectable crawlers obey them, others might not.
• Some user-triggered fetchers are documented as possibly not following robots.txt.

Verify that bots are who they claim to be#

Anyone can put "GPTBot" in a user agent string. Before you draw conclusions from your logs, or before you allow a bot through a firewall, verify it:

• Published IP ranges. OpenAI, Anthropic, Perplexity and Common Crawl publish JSON lists of the IP addresses their crawlers use, linked from the documentation pages above. Match requests against these ranges.
• Reverse DNS. Google documents a two-step check for its crawlers: a reverse DNS lookup on the IP should return a Google-owned hostname, and a forward lookup on that hostname should return the same IP. Common Crawl documents reverse DNS for CCBot as well.
• Enforcement at the edge. If a bot ignores your robots.txt, block it at the firewall or CDN level by IP or behavior. Robots.txt can't do that for you.

Also, don't use robots.txt to hide sensitive pages. The file is public, so listing a path advertises it. Protect private content with authentication.

robots.txt vs llms.txt#

llms.txt is a separate proposal that gives language models a curated guide to your site. It doesn't grant or deny access. Access control stays in robots.txt and at the server. If you want a guide file, the LLMs.txt Generator can draft one or check an existing one, and what is llms.txt explains its status honestly.

Quick checklist#

• Decide your policy separately for training, search and user-triggered agents.
• Give each AI agent its own group, and repeat the shared Disallow rules in it.
• Test the file against each user agent after every change.
• Verify bots by IP range or DNS before trusting user agent strings.
• Review operator documentation periodically. New agents appear and descriptions change.