AI training crawlers

Your robots.txt has a Disallow group that covers your homepage (for example Disallow: /) and applies to GPTBot, ClaudeBot, CCBot, Google-Extended or Applebot-Extended. Cloudflare's managed robots.txt adds groups like these when you turn it on.

In search engines

No effect on Google Search.

In AI assistants

Blocking GPTBot or ClaudeBot keeps your pages out of future training data. ChatGPT search and Claude search use different crawlers. Google-Extended also controls whether the Gemini app uses your pages in answers.

We read your robots.txt rules for the crawlers GPTBot, ClaudeBot and CCBot, and for the robots.txt tokens Google-Extended and Applebot-Extended, as they apply to your homepage.

Your choice. To opt out of training only, add a Disallow group for each training token and leave the * group open.

• CM-104 Search and AI search crawlers allowed: Reads the same robots.txt for the search crawlers, which count toward your score. Blocking a training crawler here never changes that check.
• CM-903 Firewall treatment of AI crawlers: Tests what your CDN or firewall does when an AI crawler asks for your homepage. This check reads only robots.txt.

• OpenAI: Overview of OpenAI crawlers (opens in a new tab)
• Anthropic: Does Anthropic crawl data from the web? (opens in a new tab)
• Google: Google's common crawlers (Google-Extended) (opens in a new tab)