Search and AI search crawlers allowed

Your robots.txt has a Disallow rule that covers your homepage (for example Disallow: /) and applies to Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot or PerplexityBot, through that bot's own group or through the * group. Or the request for your robots.txt fails, or your server answers it with a server error.

In search engines

A Googlebot or Bingbot block removes you from Google, Bing and everything built on them.

In AI assistants

ChatGPT search, Claude and Perplexity each use their own search crawler. If robots.txt blocks it, that assistant can't cite you.

We read your robots.txt the way crawlers do and test whether each crawler may fetch your homepage: a bot follows its own group if it has one, otherwise the * group, and no rule at all means allowed. Mentioning a bot's name doesn't count; only the rules do. We read up to 1,000 rules for each crawler. A missing robots.txt (a 404) allows every crawler. If the request for robots.txt fails (for example a timeout, a DNS failure, a refused or reset connection, a certificate error or too many redirects) or your server answers with a 5xx error, the check fails, because crawlers read that as "crawl nothing" until it's fixed. Blocking Googlebot or Bingbot from your homepage fails the check; blocking only the AI search crawlers from it earns one of its two points. ChatGPT-User, Claude-User and Perplexity-User aren't scored: they fetch a page when a person asks an assistant to open it.

You don't need a line per bot. 'User-agent: *' with 'Allow: /' lets every crawler in. Only add a named group to block a bot, and remember a bot with its own group ignores your * rules.

• CM-901 AI training crawlers: Reads the same robots.txt for the training crawlers GPTBot, ClaudeBot and CCBot and the tokens Google-Extended and Applebot-Extended. Those never change your score; the search crawlers in this check do.
• CM-903 Firewall treatment of AI crawlers: Tests whether your CDN or firewall blocks an AI crawler whatever robots.txt says. This check reads only the robots.txt rules.
• CM-105 Indexable and quotable: Looks at the noindex and nosnippet rules on the page itself. This check asks whether crawlers may fetch the page in the first place.
• CM-101 Content in the HTML: Counts the readable words in your homepage's HTML. This check asks whether robots.txt lets search crawlers fetch that page at all.

• RFC 9309: Robots Exclusion Protocol (opens in a new tab)
• OpenAI: Overview of OpenAI crawlers (opens in a new tab)
• Anthropic: Does Anthropic crawl data from the web? (opens in a new tab)
• Perplexity: Perplexity crawlers (opens in a new tab)
• Google: Robots.txt introduction and guide (opens in a new tab)