AI crawler analytics: see which AI crawlers fetched which pages

    AI crawler analytics shows which AI crawlers, from OAI-SearchBot and ChatGPT-User to GPTBot, ClaudeBot and PerplexityBot, requested which pages of your site, and when. cite-met reads every request at the edge for the sites it hosts, so those fetches sit next to your human and search traffic instead of being filtered out.

    Measure: traffic, AI-crawler intelligence, and a live AI-visibility score.
    See the whole platform

    Last updated

    The question it answers

    Which AI crawlers fetched which of my pages, and did that change after a release?

    What it is

    cite-met classifies every visit as Human, SEO (search crawlers), AEO (AI crawlers), or Bots (noise, filtered by default), and breaks the AEO stream down by crawler: ChatGPT, Perplexity, Claude, Cohere and other AI crawlers. A per-day chart overlays deployment markers so you can tie every release to how AI crawlers respond. It records fetches. It does not show whether an answer cited you.

    Traffic analytics with Human, SEO, and AEO visitor types charted separately

    Traffic analytics with Human, SEO, and AEO visitor types charted separately (example data)

    Per-crawler visibility

    ChatGPT, Perplexity, Claude, Cohere and other AI crawlers, broken out.

    Human / SEO / AEO / Bots

    Four clean streams, with junk bot traffic filtered out by default.

    Deploy markers

    Tie each release to how AI crawling responds.

    A concrete example

    In the example project above, 2,525 visits in seven days split into 1,563 Human, 574 SEO and 388 AEO, with 152 bot requests filtered out. Switch from the Category view to the Crawler view and the AEO visits split by crawler. The chart marks the project's two deployments, so you can compare crawler activity before and after each one. This counts fetches, not citations: it shows that a crawler asked for a page, not that an answer used it. The figures are example data.

    Why it matters

    Google Analytics 4 leaves out known bots automatically, and most standalone counters only run in a browser, so AI crawler visits never reach their reports. Here they're a first-class column: you see which AI crawlers asked for which pages, and when. AI-crawler analytics need the site served through cite-met, because a crawler never runs a browser script. It's included on every plan, from the $300 a month Local plan.

    Guides

    More of the platform

    Ranking isn't enough. Get cited.

    Scan your site free, then host, measure, and improve it all in one place.

    See pricing

    AI-Crawler (AEO) Analytics: FAQ

    How does cite-met know which AI crawlers visited?
    It identifies AI crawlers by their user agents at the edge. That includes the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), the ones that fetch a page when a person asks about it (ChatGPT-User, Claude-User, Perplexity-User), and training crawlers such as GPTBot and ClaudeBot. It separates them into the AEO stream, distinct from human visitors and classic search crawlers. Google-Extended is a robots.txt setting rather than a crawler, so it never appears in logs.
    Is this different from regular bot traffic?
    Yes. Noise bots are filtered out by default. AI crawlers, the ones that read your content for AI search or fetch it when someone asks, are elevated to a first-class traffic type you can measure.
    Which AI crawlers matter, and what does each one do?
    Three kinds show up. AI search crawlers build the index an assistant searches: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude) and PerplexityBot (Perplexity). User-triggered fetchers load a page because a person asked about it: ChatGPT-User, Claude-User and Perplexity-User. Training crawlers collect pages for future models: GPTBot (OpenAI) and ClaudeBot (Anthropic). Blocking a training crawler in robots.txt doesn't block the search crawlers, so decide each one on its own.
    How can I see if GPTBot or ChatGPT crawls my site without cite-met?
    Search your server or CDN access logs for the user agents. On a typical Linux server, run: grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot" access.log. Then look at the status codes: 200 means the crawler got the page, 403 means something blocked it, 429 means it was rate-limited, and 404 means it asked for a page that isn't there. Check that your robots.txt doesn't disallow the crawlers you want, and that a firewall or bot-protection rule isn't returning 403 to them. Our free scan checks the robots.txt side for you.
    Can it tell me whether ChatGPT cited my page?
    No. A fetch shows a crawler asked for a page, not that an answer used it. For citations, use the AI Performance report in Bing Webmaster Tools and the Generative AI performance report in Google Search Console, or ask a fixed set of buyer questions on a schedule and record who gets named.