How Accurate Is AI Crawler Analytics? We Audited Ours

Less accurate than most dashboards suggest, ours included. Counts labelled only by user agent overstate real assistant activity, and AI-visit counts can disagree with each other by two to one. In 90 days of data from 24 sites we host, we found:

- Hundreds of requests carrying ChatGPT, Claude, Perplexity or Mistral assistant names while asking for password and cloud-key files. - Homepage fetches arriving at a steady weekly rate that looks more like a monitoring tool than like people. - Two of our own AI-visit counts for the same sites that differ by roughly two to one, over slightly different windows.

Most AI crawler analytics, ours among them, label a request by its user-agent string, and any script can send any user-agent string. The numbers are still useful once you know which parts can be checked and which can't.

This post is an audit of our own measurement, so it uses every site on the platform, our own and new ones included. For market figures, such as which crawlers visit small-business sites and how many visits AI assistants send them, see the [Three Impact AI Search Observatory](https://threeimpact.com/ai-search-observatory). It is published by the same team from the same platform, with a stricter sample of 12 established client sites, so its totals differ from the ones here.

The sample is 24 live websites hosted on cite-met between 10 August and 8 November 2026: 10 local-service businesses, 8 B2B and software companies, and 6 publishing sites. Nine of the 24 are our own, the 6 publishers and 3 of the B2B sites. There are no online shops in it. Fifteen of the sites were live for at least 60 of the 90 days, and the other nine launched during the window. The sites were all built by one web studio that works with us, and are mostly Canadian, so treat this as a small, non-random sample.

What we had: crawler and assistant requests counted at the edge and labelled by user agent, the top 50 paths per site, weekly totals, and visits recorded by our analytics script, Pulse. What we didn't have: client IP addresses, response status codes, or timestamps finer than a week. We don't name any site, and we give traffic as shares and ratios rather than counts.

A "fetch" in this post is one request with an assistant's user agent. One question can cause several, a cached answer causes none, and none of them tells you whether the answer cited you. Claude-User made up a third to half of all fetches on some sites, and part of that came from our own research during the period, so read Claude figures as high.

The clearest cases are a few hundred requests that carried an AI-assistant user agent while asking for files no assistant answering a question would need: cloud credentials such as `.aws/credentials`, `.env` files, private SSH keys, server admin endpoints, and Anthropic API key files such as `.anthropic/config.json`. They went to three local-service sites, under the ChatGPT-User, Claude-User, Perplexity-User and MistralAI-User names.

Without IP addresses we can't say who sent them. What the paths prove is that they weren't assistants answering a question. These look like the internet-wide sweeps every public site gets, and three sites isn't enough to say local businesses are singled out.

These sites' assistant traffic came in bursts: on each of the three, a single week, starting either 24 August or 7 September, carried between a third and three-fifths of all the assistant fetches the site received in 90 days.

The true count is probably higher, for two reasons. Our data keeps only each site's top 50 paths, and on these three sites more assistant fetches fell outside that list than the probes we could see. And a scanner that asks for ordinary pages looks exactly like an assistant in user-agent data. A fourth site, a B2B company, received about 70% of its assistant fetches for the whole 90 days in those same two weeks, all to ordinary pages. That fits the pattern, and we can't prove it either way from user agents alone.

Never allowlist a bot by user agent alone, in a firewall rule or anywhere else, because that lets every scanner through under a borrowed name. And check what those requests got back. If your site answers every unknown path with a 200 and your homepage, which many single-page apps do by default, scanners can't tell a hit from a miss and your logs fill with false "pages", so return a real 404 for paths that don't exist.

No. Many fetches come from tools and from your own team. OpenAI documents [ChatGPT-User](https://developers.openai.com/api/docs/bots) as the agent it uses when a ChatGPT user's request needs a page. Plenty of those users aren't prospects. One local-service site in our sample had weekly assistant fetches that stayed within about 12% of each other for seven straight weeks from 24 August, then settled about an eighth lower, and 81% of all its fetches were ChatGPT-User requests for the homepage. Weekly totals can't show a schedule, but a series that flat on a small local business is what you would expect from a scheduled process, and AI-answer tracking tools are a likely source. We can't rule out that some of it comes from cite-met's own prompt checks.

Tracker fetches are genuine. They come from the vendor's own servers and pass an IP check, because a tracking tool asking ChatGPT a question makes ChatGPT open your page exactly as a buyer would. Verified means the request came from OpenAI, not that a person was behind it.

Your own team counts too. Every time someone in your company asks ChatGPT or Claude about your business and the assistant opens your site, that is a fetch. Research and testing during a launch can move the weekly number on a small site.

Treat fetch counts as an upper bound on attention. One check you can run: if a single page takes nearly all of one agent's fetches at a steady weekly rate, suspect a tracker before you suspect demand.

Mostly because they are built differently. For the same 24 sites, a second AI-visit count in our portfolio reports, built separately from Pulse, our analytics script in the browser, came out nearly twice as high as Pulse's, even over a slightly shorter window. On one local site it was more than 15 times higher. The two don't classify visits the same way, and on some sites Pulse may have been installed after the window began, so part of the gap is method rather than error. We haven't pinned down the rest, which is why we treat Pulse as a floor. Lost referrers make both counts lower, so they don't explain the gap, but they do mean both counts miss visits. A click can lose its source in several ways. Cloudflare notes that [traffic from native apps may not include a Referer header](https://developers.cloudflare.com/radar/glossary/), in-app browsers often strip it, and a site's own `Referrer-Policy` can cut it to the domain or remove it. A visit with no referrer is filed as direct.

Upstream breaks add to it. ChatGPT links appear to have lost both the `utm_source=chatgpt.com` tag and the referrer for several days in early October 2026. [Search Engine Roundtable reported](https://www.seroundtable.com/utm-parameters-missing-chatgpt-42299.html) data from Adobe's Cedric Huesler showing ChatGPT visits falling while direct visits rose on the same pages from 9 to 13 October, until the tag came back on 15 October. Anyone comparing October with September needs a note on that week. We cover setting up ChatGPT visit tracking in [how to track ChatGPT traffic](/blog/post/how-to-track-chatgpt-traffic).

Google Analytics 4 adds two blind spots of its own. Its [AI Assistant channel](https://support.google.com/analytics/answer/9756891?hl=en) leaves out Google's AI Overviews and AI Mode, which stay inside Organic Search. And most AI crawlers never run the GA4 tag at all, since, as [Vercel found](https://vercel.com/blog/the-rise-of-the-ai-crawler) in December 2024, none of the major ones rendered JavaScript. For the bots that do, GA4 [removes known bots automatically](https://support.google.com/analytics/answer/9888366?hl=en), with no setting to turn that off and no count of what it removed.

The opposite error shows up in server-side counts. One site in our sample logged more than 700 server pageviews for every visit Pulse recorded. Pageviews and visits are different units, but a gap that size means most server pageviews are not people, so any dashboard that reports raw server pageviews, ours included, overstates human traffic by a wide margin.

Check the request's IP address against the list the vendor publishes, or run a forward-confirmed reverse DNS lookup where the vendor supports one: look up the hostname for the IP, check that it ends in the vendor's domain, then look up that hostname and confirm it returns the same IP. The Observatory found 416 requests on 10 of its 12 sites claiming to be [Google-Extended](https://threeimpact.com/ai-search-observatory#fake-google-extended-requests), a name Google never sends, which is the cleanest proof that these strings get borrowed. A user-agent string alone proves nothing, and [Google says plainly](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) that user agents can be spoofed.

| Vendor | Agents | How to verify | |---|---|---| | [OpenAI](https://developers.openai.com/api/docs/bots) | GPTBot, OAI-SearchBot, ChatGPT-User | Published IP lists: `gptbot.json`, `searchbot.json`, `chatgpt-user.json` | | [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | ClaudeBot, Claude-User, Claude-SearchBot | Published IP list: `claude.com/crawling/bots.json` | | [Perplexity](https://docs.perplexity.ai/guides/bots) | PerplexityBot, Perplexity-User | Published IP lists, one per agent | | [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) | Googlebot | Reverse DNS to `googlebot.com`, or the published IP list | | [Apple](https://support.apple.com/en-us/119829) | Applebot | Reverse DNS to `*.applebot.apple.com`, or `applebot.json` | | [Mistral](https://docs.mistral.ai/robots) | MistralAI-User | Published IP list | | [DuckDuckGo](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) | DuckAssistBot | Published IP list | | [Common Crawl](https://commoncrawl.org/ccbot) | CCBot | Reverse DNS to `crawl.commoncrawl.org`, or `ccbot.json` | | [Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) | meta-externalagent, meta-externalfetcher | IP ranges announced by Meta's network, AS32934, which Meta says change often |

Google's AI products don't appear in a crawler log as such. Google-Extended is a robots.txt token with no user agent of its own. Google says that crawling is done with its existing user agents, so no log can separate Google's AI use from search crawling.

Done by hand or in a firewall rule, the IP check has a few traps:

- Behind a CDN or proxy, check the real client IP from the forwarded header your CDN sets, not the address of the CDN itself. - Fetch the vendor's list fresh, at least daily, because ranges change. - Match each list only to its own agent. A ChatGPT-User request from a GPTBot range is still a mismatch. - Don't treat any address in a vendor's cloud provider as theirs. Scanners rent servers from the same clouds.

A request that claims to be one of these agents and comes from outside the vendor's current list is unverified. Recheck against a fresh copy, and if it still doesn't match, treat it as an impostor. Anthropic also warns that blocking by IP "may not work correctly or persistently", so prefer robots.txt where the vendor honours it. On Cloudflare's free plan, [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) identifies crawlers by user agent only, and paid plans add Cloudflare's bot detection on top.

Robots.txt behaviour is a second clue. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores" them, while Anthropic says Claude-User follows them. A Claude-User request to a path your robots.txt blocks for Claude-User, or for every agent, deserves a closer look, allowing for the fact that vendors cache robots.txt for a while after you change it. Don't list secret paths in robots.txt to hide them. It is a public file, and it tells scanners where to look.

Trends in verified crawler traffic are the most dependable figures on an AI crawler dashboard, because the vendor's own IP list backs them. Treat everything labelled by user agent alone as a ceiling, and read the paths behind any spike before you report it.

For visits, compare two methods before trusting either, and mark known breaks like the October ChatGPT tagging gap in your reports. For leads, count from your forms and bookings, not from referrers. In our sample, Pulse saw fewer than one in ten of the leads that came through forms and bookings inside a visit it had tracked, from any source. Booking tools on other domains probably account for some of that gap, and it still shows how little of the lead trail referrer data captures on small sites.

Our [AI crawler analytics](/features/ai-crawler-analytics) labels crawler and assistant requests by user agent and shows the pages each agent asked for, which is how we found the probes in this post. It doesn't yet check each request against the vendors' IP lists, so treat its counts as the ceiling described above. Pulse counts visits by referrer, so it has the referrer gaps described above too, and we report it as a floor. Leads come from your own forms and bookings.

If that split is useful for your site, the [plans page](/pricing) lists which plan includes it. For the tracking side of the same question, see [how many prompts and runs you need](/blog/post/how-many-prompts-ai-visibility).

*Data: 24 sites hosted on cite-met, 10 August to 8 November 2026, aggregated and anonymised. Visit, lead and fetch volumes are given as ratios, not counts.*