AI Crawlers and robots.txt: GPTBot, ClaudeBot, Google-Extended and a Policy You Can Defend

Most robots.txt files were written for a web with one crawler that mattered. Today one article can be fetched by a bot collecting training data, a bot building an AI search index, and a bot acting for one person who asked about your page — often from the same company, under different names, with different consequences when you say no.

"Block AI" and "allow AI" are the wrong unit of decision. The right unit is purpose. This guide maps who is crawling and what each block costs, then gives you three templates you can defend in a meeting. It goes much deeper than the crawler-policy section of our GEO and AEO guide.

Three kinds of AI access

AI agents fall into three classes, with opposite economics:

1. Model-training crawlers collect pages that may become training data. Value flows one way: your content improves the model, and nothing links back.
2. AI search indexers build the index that AI answers cite from. Block them and you stop being a candidate for citation.
3. User-triggered fetchers open one page because a person asked the assistant to. They are not crawling; they are that user's browser at one remove.

Agent	Operator	Purpose	If you block it
GPTBot	OpenAI	Model training	Future pages left out of training; ChatGPT search unaffected
ClaudeBot	Anthropic	Model training	Future pages left out of training
CCBot	Common Crawl	Open web archive, widely used as training data	You leave a corpus many model builders use
meta-externalagent	Meta	AI training and product indexing	Future pages left out of those uses
Bytespider	ByteDance	Crawling linked to ByteDance's AI products	Left out if it complies; operators report mixed results
OAI-SearchBot	OpenAI	ChatGPT search index	Not shown as a source in ChatGPT search
Claude-SearchBot	Anthropic	Claude search index	Less visible in Claude's search answers
PerplexityBot	Perplexity	Perplexity search index	Unlikely to be cited in Perplexity
ChatGPT-User	OpenAI	Fetch on a user's request	ChatGPT may not open your page for that user*
Claude-User	Anthropic	Fetch on a user's request	Claude may not read your page for that user*
Perplexity-User	Perplexity	Fetch on a user's request	Perplexity may not open it for that user*

* Only where the vendor applies robots.txt to user-initiated requests at all — see below.

So one vendor can be welcome in one class and refused in another: blocking GPTBot while allowing OAI-SearchBot is coherent, and it is where most citation-minded publishers land. And every effect is forward-looking: robots.txt governs future fetches, not data already collected or models already trained.

Control tokens that never appear in your logs

Some names in an AI policy are not crawlers but product tokens: robots.txt names that a vendor's ordinary crawler checks before using content for a specific purpose.

Token	Who fetches	Controls	Does not control
Google-Extended	Googlebot	Gemini model training and grounding in Gemini apps and Vertex AI	Google Search inclusion or ranking, per Google
Applebot-Extended	Applebot	Training of Apple's foundation models	Apple search features such as Siri and Spotlight

The trap: AI Overviews and AI Mode are part of Google Search, so Google-Extended does not govern them. Normal Search controls do — nosnippet, data-nosnippet, max-snippet, ultimately noindex — and they shape your classic snippets too. At the time of writing there is no switch that removes you from AI Overviews alone; check Google's current documentation.

You will never see these tokens in an access log. And never enforce them at a firewall: that traffic is Googlebot and Applebot, and blocking it takes you out of search.

Vendor behaviour differs, and it changes

The names are accurate at the time of writing; the rules around them are less stable.

• User-triggered fetchers are the grey zone. Some vendors say they honour robots.txt for these agents; others state it may not apply, because a human initiated the request. Treat a block on a -User agent as a request, not a guarantee.
• Agents get split, renamed and added. Old policies miss new agents.
• Not every crawler announces itself. robots.txt only speaks to bots that choose to listen.

Bookmark each vendor's crawler documentation, re-read it quarterly, and date-stamp your robots.txt.

robots.txt mechanics that trip people up

robots.txt is standardised as RFC 9309. Most AI-policy failures are parsing mistakes, not strategy mistakes.

A crawler obeys one group: its own

A crawler follows only the group whose User-agent matches its name. User-agent: * is a fallback for bots without their own group, not a base layer others inherit. The most common bug looks like this:

# WRONG: GPTBot ignores the * group entirely
User-agent: *
Disallow: /dashboard/
Disallow: /account/

User-agent: GPTBot
Allow: /

GPTBot may now crawl /dashboard/ and /account/. Repeat every restriction in every named group:

# RIGHT: each named group carries the full set of restrictions
User-agent: *
Disallow: /dashboard/
Disallow: /account/

User-agent: GPTBot
Disallow: /dashboard/
Disallow: /account/

Allow: / was never needed; anything not disallowed is allowed. Pasted under your rules, any snippet granting a bot Allow: / silently exempts it from all of them. Stacked User-agent lines above one rule set form a single group, which keeps shared rules in one place.

Paths are prefixes, and the longest match wins

• Disallow: /app also blocks /apple-pie. Write /app/ unless you mean the prefix.
• * matches any characters and $ anchors the end: Disallow: /*.pdf$.
• When Allow and Disallow both match, the longer rule wins; on a tie, Allow wins.

The status code of robots.txt is a directive

Under RFC 9309, a 404 means "no rules" and a 5xx means "assume everything is disallowed". A broken deploy can open or close your whole site. Crawlers may also cache the file for up to 24 hours.

It is a sign on the door, not a lock

robots.txt is advisory: compliant crawlers obey it; nothing forces anyone else to. Protect private areas with authentication. And the file is public, so Disallow: /internal-admin-2024/ is an advertisement.

Three policy templates

Swap in your own paths. Each template keeps shared restrictions in every group that needs them.

A. "Cite me, don't train on me"

For publishers who want citations without contributing training data.

# AI crawler policy - reviewed 2026-09-24
# Training crawlers and training tokens: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# AI search indexers: welcome, minus the product
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /app/
Disallow: /account/
Disallow: /api/

# Everyone else, including user-triggered fetchers
User-agent: *
Disallow: /app/
Disallow: /account/
Disallow: /api/

Sitemap: https://www.example.com/sitemap.xml

The search group duplicates * today; naming the bots documents intent. Note that Google-Extended covers grounding as well as training, so blocking it also removes you from grounding in Gemini apps. Google Search and AI Overviews are unaffected.

B. "Open editorial, closed product"

For maximum reach on public content, training included, and no crawling of the application.

# AI crawler policy - reviewed 2026-09-24
# One group for every bot: nothing to bypass
User-agent: *
Disallow: /app/
Disallow: /account/
Disallow: /settings/
Disallow: /api/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

With no named groups, no bot can be exempted by accident. To shut one crawler out, add a group with Disallow: / — a full block cannot leak. To treat one partly differently, copy every * line into its group first.

C. Full opt-out

# AI crawler policy - reviewed 2026-09-24
# Opt out of AI training, AI search and user-triggered AI fetches
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /app/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

What you lose: citations and referrals from ChatGPT search, Claude and Perplexity, and those assistants' ability to open your page when a user asks. What you do not get: removal from AI Overviews, deletion of data already collected, or protection from bots that ignore robots.txt. If opt-out is a legal requirement, robots.txt declares it and your CDN enforces it.

Verify who is actually knocking

A user-agent string proves nothing

Any script can send User-agent: GPTBot, and scrapers impersonate known crawlers precisely because sites let them through. Confirm the vendor before blaming it.

Reverse DNS, forward-confirm, or published IP ranges

For Googlebot, do a reverse lookup, then a forward lookup that must return the same IP:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

OpenAI and Google publish machine-readable IP ranges for their crawlers, and some other vendors do too. Where nothing is published, treat the user agent as a claim, not an identity.

Read your logs grouped by agent

# Hits per AI agent (combined log format)
grep -oiE 'gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user|ccbot|meta-externalagent|bytespider' access.log \
  | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn

# Top paths requested by one agent
grep -i 'gptbot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Look for hits on disallowed paths (a group bug or an impostor), unfamiliar agents and bandwidth spikes, aggregated by path pattern as in the crawl budget guide. Then test the policy against your real inventory: a crawl that reads robots.txt and sitemaps — DeepAnalyzer does both — shows whether any sitemap URL, or any page you want cited, sits under a disallowed prefix.

Enforce at the edge when it truly matters

For non-compliant bots, bandwidth problems or contractual obligations, enforce at your CDN or WAF. Cloudflare, for example, offers AI-crawler blocking controls, and most bot-management products match on verified identity rather than the raw string. Check, too, that no edge setting blocks bots your robots.txt invites.

Signals that are not robots.txt

noai and noimageai

<meta name="robots" content="noai, noimageai"> is not part of any standard, and major AI crawlers have not broadly committed to honouring it. Harmless to add, unwise to rely on. Efforts such as the IETF's AI Preferences work aim to standardise usage preferences; until vendors commit, treat them as supplementary.

llms.txt

llms.txt is a 2024 community proposal from Jeremy Howard of Answer.AI: a Markdown file at /llms.txt that points language models to your key content at inference time. It is a curated map, not a permission file.

# Example Docs

> Example is an API for scheduling background jobs.

## Docs

- [Quickstart](https://www.example.com/docs/quickstart.md): Install the SDK and run a first job
- [API reference](https://www.example.com/docs/api.md): Endpoints, parameters, errors

## Optional

- [Changelog](https://www.example.com/changelog.md): Release history

It is cheap and useful for developer docs. But as far as is publicly known, no major AI search provider has committed to using it as a ranking or citation signal. Write it in an afternoon; do not expect traffic.

Measure what the policy costs and earns

AI referrals. Build an analytics segment on the referrer:

chatgpt\.com|perplexity\.ai|claude\.ai|gemini\.google\.com|copilot\.microsoft\.com

ChatGPT links often carry utm_source=chatgpt.com, so check campaign source too. Expect undercounting: app traffic lands in direct, and AI Overviews clicks look like ordinary Google organic.

Crawler volume. Monthly hits per agent, bytes served, hits on disallowed paths. Set a vendor's crawl volume against the referrals its products send you.

Readability. Not every AI crawler executes JavaScript, and an open policy is worthless if the bot gets an empty shell. Compare raw HTML with a rendered view — Strong Mode renders pages in a headless browser — and see JavaScript rendering and crawlability.

Each quarter: re-read vendor docs; count hits per agent and verify top IPs; chase every hit on a disallowed path; compare AI referrals; ask the major assistants your top ten questions and record who is cited; update the date in robots.txt.

Decision checklist

• Which content should be cited and which kept out of training is written down, with an owner
• Every named AI group repeats every restriction from *
• /robots.txt returns 200 and is checked after deploys
• Private areas require authentication; robots.txt names no secret paths
• Nobody is "blocking" Google-Extended or Applebot-Extended at the firewall
• AI Overviews exposure is a conscious Search-controls decision
• CDN bot settings agree with robots.txt
• An AI-referrer segment exists, and logs are reviewed by agent monthly
• A quarterly review is on the calendar

FAQ

Does blocking GPTBot remove my content from ChatGPT?

No. It stops future collection for training but does not remove data already collected. Appearing in ChatGPT search is governed by OAI-SearchBot, and on-demand visits come from ChatGPT-User, so decide on each separately.

Will blocking Google-Extended hurt my rankings or remove me from AI Overviews?

Google states that Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews are part of Search and follow Search controls such as nosnippet and max-snippet. The token controls Gemini training and grounding in Gemini apps.

Why is a bot crawling a path I disallowed?

Usually the bot has its own User-agent group without the restriction, so your * rules never applied. Otherwise it is an impostor. Check the group, then verify the IP with reverse DNS or the vendor's published ranges.

Should I add an llms.txt file?

If you publish developer documentation, yes: it takes an afternoon and helps assistants find the right pages. No major AI search provider has publicly committed to it as a ranking or citation signal, so do not expect a traffic change.

How often should we revisit the policy?

Quarterly, and whenever a vendor announces a new agent. Names and robots.txt handling change, especially for user-triggered fetchers, and a recurring review is cheaper than discovering drift in your logs.