See which AI crawlers read your site — and prove which ones are real
Track every AI crawler that hits your pages — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — verified against each vendor’s published IP ranges so a spoofed user-agent can’t fake its way in. Then close the gaps: map each blocked answer-engine to a concrete fix and apply it to your robots policy on WordPress in a click.
Credit card required · Cancel anytime · 7-day full access
| Page | Clicks | Status |
|---|---|---|
| / | 412 | Improving |
| /blog/ai-search-guide | 286 | Improving |
| /pricing | 204 | Healthy |
| /features/visibility | 171 | Healthy |
- Organic Search46%
- Direct32%
- Referral14%
- AI answers8%
01The blind spot
Your analytics can’t see them. Your log can’t verify them.
Analytics runs on JavaScript, and crawlers don’t execute it — so the fastest-growing readers of your site never appear. Your server log does see them, but all it records is a user-agent string: a piece of text any script can type. The category has been counting text and calling it traffic.
- Analytics tools measure people — AI crawlers never fire a pageview
- A log line “from”
GPTBotis a claim, not an identity —curl -A "GPTBot"produces one - Every unverified AI-crawler count you’ve seen is a count of strings, not visits
203.0.113.7 - - [12/Aug/2026:03:41:07 +0000] "GET /blog/website-cost-pricing-guide HTTP/1.1" 200 48213 "-" "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)"
$ curl -A "GPTBot" https://yoursite.com/
# that is all it takes to be “GPTBot” in an analytics tool
02Verified, not claimed
858 hits claimed GPTBot. 162 were telling the truth.
Last week, on a real account: 858 visits said GPTBot. SearchChamp checked each one against the source IP and OpenAI’s published ranges — 162 verified, 696 spoofed. Every one of the 78 hits claiming ClaudeBot was fake. A badge is only awarded when the address checks out, so the number you act on is the number that happened.
Verified · 162— source IP confirmed to belong to the bot’s operatorSpoofed · 696— the name said GPTBot; the source address didn’t- All 78 hits claiming ClaudeBot that week failed the check
03How the check works
Checked at the door. The raw IP is never written down.
Every AI-crawler hit is matched against the real source IP and the address ranges the crawler’s owner publishes — refreshed on a schedule, so a vendor adding IPs doesn’t silently turn real traffic into “spoof”. The check runs before the IP is hashed: verification happens at the door, and only the verdict is kept. When a crawler relies on DNS identity, a bounded reverse-DNS forward-confirm also finishes before storage.
- Real source IP vs the operator’s published ranges — a name alone earns nothing
- Ranges refresh on a schedule, so new vendor IPs don’t become false spoofs
- The raw address is discarded once the verdict exists — checked, then hashed
UA "GPTBot"
IP 203.0.113.7as it arrives
04Five outcomes, not two
Because “we don’t know” is three different things.
Most tools give you a binary: bot or not. SearchChamp reports five verdicts, because an unprovable identity isn’t the same as a fake one — and a check that hasn’t finished isn’t the same as a check that failed. When it can’t prove something, it tells you which kind of can’t-prove you’re looking at.
Unverifiablemeans the vendor publishes no ranges — not that the bot is fakePending checkresolves;Check degradedis our outage on the record, not yours- A row with no recorded capture tier shows no badge at all — never a guess
05Three jobs, one company
Training, indexing, or fetching live — the job matters more than the name.
OpenAI alone runs GPTBot to gather training data, OAI-SearchBot to index pages for citation, and ChatGPT-User to fetch your page live when someone asks about you. Block the wrong one and you vanish from live answers while still feeding the training set. SearchChamp groups every crawler by what it actually does.
Breakdown by crawler
Last 7 days, grouped by what each bot does — and how many of its visits we could verify.
Training crawlers
Bots that fetch your content to train AI models.
Search-index crawlers
Bots that index your pages so AI assistants can cite them in answers.
User-triggered fetchers
Bots that fetch a page live when a user asks an AI assistant about it.
06Every crawler, named
The full roster, by real registry token.
Every crawler is tracked by the token your log actually shows, vendor beside it — from GPTBot and ClaudeBot down to meta-externalfetcher and CCBot. And when something crawls you that isn’t in the registry, it lands in Unknown AI: recorded, never dropped. An unrecognised bot is a finding, not noise.
- Real tokens like
OAI-SearchBot (OpenAI)— the string your log actually shows - Sibling crawlers stay separate:
Claude-Useris notClaudeBot Unknown AIcatches the rest — recorded, never dropped
07Capture, four ways
Something has to see the request. Pick what fits your stack.
Crawlers don’t run JavaScript, so capture happens where the request lands: edge middleware on Vercel or Next.js, a Cloudflare worker, the SearchChamp WordPress plugin, or a shipper that posts your existing access-log lines. The beacon is non-blocking and adds no page-speed cost; the log route changes nothing on your pages at all.
- Ingest URL + secret token, sent in
X-Ingest-Token— rotate it if it leaks - Non-blocking by design — detection never sits in the request path
- Every row is stamped
Beacon,Access logorEdge worker
// middleware.ts — detects AI-bot user-agents, non-blocking export function middleware(req) { reportAiBot(req) // fire-and-forget beacon return NextResponse.next() // page speed: untouched }
Sent in the X-Ingest-Token header. Keep it secret — rotate it if it leaks. Verify: request any page with a user-agent containing “GPTBot”.
08An empty list that explains itself
“No bots yet” and “capture broken” are different facts.
A quiet dashboard is only reassuring if capture is provably alive. SearchChamp self-tests the tracker and tells you which state you’re in: working, not working, or not yet tested. An empty list under a passing self-test means no bots came — and it says so in those words.
- A dated self-test proves the pipe is open before you trust a zero
- “Too long since the last self-test” is called out, not papered over
- Until the first test runs, the empty state says exactly what it can’t claim
09Windows, honestly
The window you see is the window you actually have.
Every view runs on a 7 / 30 / 90-day toggle, with your plan’s real retention beside it: Starter keeps 7 days of crawl history, Pro 90, Agency the full record. A locked window says why it’s locked. And if your plan can’t be determined, SearchChamp falls back to the shortest window — it never flatters you with data it shouldn’t show.
- Locked windows read “Available on higher plans” — no silent empty charts
- Unknown plan? Fail closed: the most restrictive window, never the longest
- Past the daily event cap, a visible sampling notice — not silent truncation
Available on higher plans — your Starter plan includes a shorter history window.
UNKNOWN PLAN → FALLS BACK TO THE MOST RESTRICTIVE WINDOW. A LONGER ONE IS NEVER LEAKED BY ACCIDENT.
10Crawled vs cited
Your three most-crawled pages can never be cited.
The reconciliation most tools never run: pages verified AI crawlers fetch, against pages AI engines actually cite. On this account the top three crawled targets are an image endpoint, robots.txt and the sitemap — 2,839 fetches of things no answer will ever quote. Meanwhile one page is being cited without a recent verified crawl at all.
Crawled vs. cited
How the pages AI crawlers fetch line up with the pages AI engines actually cite.
Quoted without a recent verified crawl — the engine is answering from what it already has. Check the page stays reachable and fresh.
Reconciles pages verified AI crawlers fetched against pages AI engines cited. “Crawled, never cited” pages are read but not quoted — improving extractability raises (not guarantees) citation odds. “Cited, never crawled” pages are quoted without a recent verified crawl — check they stay reachable and fresh.
11A floor, not a ceiling
Crawl volume is loud. Attributable sessions are a floor.
Verified AI-bot crawl volume against real AI-referred sessions — with the caveat most tools quietly omit: many AI apps strip the referrer, so GA4 files those visits under “(direct)”, and Bing Copilot can’t be told apart from Bing organic at all. SearchChamp filters to genuinely attributable AI sources and labels the count for what it is.
- GA4 filtered to AI-assistant sources — not the whole “Referral” channel
Measured= reliable UTM/referrer ·Referrer-only= bare host, likely undercounted- The undercount is printed on the panel, not buried in the docs
Source: GA4 (AI-source filtered) + verified crawler logs · As of Aug 14, 2026
MEASURED — reliably carries an attributable UTM/referrer · REFERRER-ONLY — GA4 only sees a bare referring host; likely undercounted
12From crawl to conversion
Crawled → Cited → Visible → Clicked → Converted.
The full connected funnel — and the three honest states a stage can be in. No source connected: a specific connect CTA, never a fabricated 0. Source connected but the live pull failed: “Temporarily unavailable” — a different thing. And a low-sample stage carries its confidence interval, so a ratio inside the error bars reads “Within noise” instead of a confident arrow.
- Real 30-day values:
Crawled 3,719·Cited 7·Visible 2.0% - “Connect GA4 to see clicks” is a state, not an error — and never a fake zero
- No conversion rate across Visible: it’s a share-of-voice %, not a count
From crawl to conversion
How visibility with AI crawlers turns into citations, clicks, and revenue.
13Can they even reach you?
Everything above is measured. This is probed — and labelled.
The verified log shows what really happened. The Access Guard answers the forward-looking question — could each answer engine reach you if it tried? — by probing your site with each bot’s user-agent. A probe is a forecast, not a verdict, so every single row carries a heuristic tag and a confidence level, and the evidence says exactly how it was measured.
AI Access Score 100/100. No AI crawlers appear to be blocked. An llms.txt file is present. Probe results are heuristic: measured with a bot user-agent from SearchChamp infrastructure, not the crawler’s verified IP.
AI answer & search
Crawlers that power AI answers and cite sources (ChatGPT, Claude, Perplexity, …)
EVIDENCE — PERPLEXITYBOT
bot user-agent served 200 with content. Heuristic: measured from SearchChamp infrastructure with a bot user-agent only. Real crawlers are verified by IP/DNS/Web-Bot-Auth against the vendor’s published ranges, so this reflects how your site responds to that user-agent, not a guaranteed verdict for the real bot.Classic search
Traditional web-search crawlers (Google, Bing)
Model training
Crawlers that fetch content to train AI models.
14Fix it, then re-check
Every fix shows its diff. Every “done” gets re-scanned.
Findings become fixes you can read before they happen: the proposed robots.txt directives and the exact resulting file. Apply to WordPress in one click, get steps for what can’t be automated — Cloudflare rules included — and marking a fix done queues a re-scan to verify it took. All under a standing reminder: robots.txt is advisory, and “Not addressed” was never an allow.
- Three directive states:
Disallowed·Allowed·Not addressed— never flattened to two - “Applied — pushed to your WordPress site.” Or honestly: no live connection to push to yet
- Publishing a preference is stated as a preference — honoured by well-behaved bots, not enforced
Fix access issues
Turn findings into fixes — apply what can be automated, get steps for the rest.
Publish an explicit allow for answer-engine crawlers
PROPOSED ROBOTS.TXT DIRECTIVES → RESULTING ROBOTS.TXTUser-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml + User-agent: OAI-SearchBot + Allow: / + User-agent: PerplexityBot + Allow: /
AI Crawler Analytics, answered.
Every AI crawler hit on your site — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — classified as verified, spoof, unverifiable, or pending, so you can see which answer engines are really reading your content instead of guessing from a user-agent string.
A bot operated by an AI company that fetches web pages so its models and answer engines can use them — either to train on, or to retrieve a page while answering someone’s question right now. It is the AI-era equivalent of Googlebot: if it cannot reach and read your page, that engine has nothing of yours to cite.
Every crawler hit is matched against the real request IP and checked against the crawler owner’s published address ranges. A bot name alone never earns a verified badge — a spoofed user-agent that does not come from the real IP range shows as spoof, not verified. On a real account in one recent week, 696 of 858 hits claiming to be GPTBot were spoofed.
GPTBot crawls the web broadly to gather content for training. ChatGPT-User is the fetch that happens when someone asks ChatGPT a question and it retrieves your page to answer it right then. They are separate user-agents you can allow or block independently — and the distinction matters, because blocking the second is what removes you from live answers, while blocking the first only affects training. SearchChamp reports them separately rather than collapsing both into “OpenAI”.
Something has to see the request. Choose whichever fits: a few lines of edge middleware on Vercel or Next.js, a Cloudflare worker, the SearchChamp WordPress plugin, or a shipper that posts your existing server access-log lines. The beacon is non-blocking and adds no page-speed cost, and the access-log route changes nothing on your pages at all.
Yes — each blocked answer engine is mapped to a concrete fix. You can see the proposed robots.txt directives and the resulting file before anything happens, apply it to WordPress in one click, and a re-scan is queued to confirm it took effect. For anything that cannot be automated, such as a Cloudflare rule, you get the steps. Worth knowing: robots.txt is advisory, not enforcement — well-behaved crawlers honour it, and nothing forces them to.
Verified crawler tracking is part of the core platform on every paid plan, not a paid add-on. What differs is how much crawl history is retained: Starter keeps 7 days, Pro 90 days, and Agency the full record. See the pricing page for what each plan costs.
No. Crawler verification runs continuously as part of monitoring a connected site rather than being billed per check, so you are not choosing between watching closely and controlling spend.
Know exactly which AI engines read — and cite — your site
7-day free trial · credit card required · cancel anytime