Firewall treatment of AI crawlers

Your robots.txt allows AI crawlers, but your server logs or CDN dashboard show them getting 403, 429 or 503 responses while people get your pages.

In search engines

No effect on Google Search.

In AI assistants

Some CDNs block AI crawlers by default, regardless of robots.txt.

We fetch your homepage once more, identifying as OpenAI's search crawler. Firewalls that check whether a crawler is genuine may block our test even when they'd let the real one in, so treat a block as a prompt to check your settings.

In your CDN or firewall settings, check the AI-crawler or bot rules (on Cloudflare: Security → Bots, and AI Crawl Control).

• CM-104 Search and AI search crawlers allowed: Reads what your robots.txt allows. This check tests what your server does when an AI crawler actually asks.
• CM-101 Content in the HTML: Counts the words in the page our scanner's normal request gets. This check compares the status code an AI crawler's request gets with that one.
• CM-901 AI training crawlers: Reads your robots.txt rules for the training crawlers and tokens, such as GPTBot and Google-Extended. This check tests what your server does when an AI crawler asks for your homepage.

• OpenAI: Overview of OpenAI crawlers (opens in a new tab)
• Cloudflare: AI Crawl Control (opens in a new tab)