What is Crawled but Not Cited?
By SearchChamp teamUpdated
Crawled but not cited describes a page that AI crawlers verifiably fetch — the requests appear in server or CDN logs under agents such as GPTBot, ClaudeBot, PerplexityBot or Google-Extended — but which never appears as a credited source when an assistant answers the questions that page exists to answer. It is a distinct diagnosis from “not crawled” and from “not indexed”. The engine can reach the content and has chosen not to use it, which narrows the possible explanations sharply: the page usually buries its answer, states it only implicitly, hides it behind interaction, or offers nothing specific enough to quote. Because the hardest part of the pipeline is already working, this class is generally the cheapest AI visibility gap to close.
Crawled but Not Cited in context
The vocabulary emerged as AI visibility tooling matured through 2025 and 2026 and vendors converged on a taxonomy of per-answer outcomes — roughly cited, synthesised without attribution, crawled but not cited, and ignored entirely. The distinction became practical once two data sources could be joined: AI crawler hits from your own access logs, and citation records from tracked answers. Neither half is sufficient alone. Crawler logs prove reachability but say nothing about whether the content was useful; citation tracking proves absence but cannot tell you whether the cause was retrieval or quality. Joined, they separate a plumbing problem from a content problem — and they disagree with the intuition of most teams, who assume they are not being crawled when the logs usually show the opposite.
Example
A payments company has a thorough 2,000-word page on marketplace payouts. Its access logs show repeated fetches from GPTBot and PerplexityBot over the past month. Yet when the tracked question “how do marketplace payouts work for split payments” is asked, the assistants answer using a competitor’s documentation page and never mention the company. Reading the page explains it: the definitional answer appears in the eleventh paragraph, after a narrative introduction; the concrete mechanics sit inside a diagram image with no text alternative; and the summary that would be quotable is inside a collapsed accordion. The remedy is not more content or more links — it is moving the direct answer to the top, expressing the mechanics as text, and un-collapsing the summary, so that the crawler that already arrives has something extractable to take away.
Why it matters in 2026
Teams routinely respond to poor AI visibility by commissioning more content, when their existing pages are being read and passed over. Separating crawled-but-not-cited from never-crawled changes both the diagnosis and the budget: one is a rewrite measured in hours, the other is a publishing programme measured in months. It is also the class with the clearest verification path, because you can compare the same tracked question before and after an edit to a page you know the engine already visits. Treat the comparison as directional — a single re-run is one sample, and engines change independently of your site — but it is far more informative than watching an aggregate score.
Related terms
Common questions about Crawled but Not Cited.
It is generally better news, because the harder half of the problem is already solved. Not being crawled points at robots.txt rules, a blocked user agent, an orphaned URL, or a rendering issue — infrastructure work. Crawled but not cited points at the page itself, which is usually a rewrite you can do this week. It is also easier to verify afterwards, since you already know the crawler visits that URL.
The usual reasons are extraction problems rather than quality judgements. The direct answer is buried below a long introduction, the substance lives in an image or a chart with no text equivalent, the quotable summary is inside a collapsed element, the claims are vague enough that nothing is safely quotable, or a competing page states the same thing far more plainly. Assistants prefer passages they can lift with low risk of being wrong, so an unhedged, specific, early sentence beats a well-argued page that never quite commits.
It cannot be promised. Making the answer explicit, early and extractable removes a known obstacle, and it is a reasonable input to test — but engines weigh source authority, freshness, and what competitors published, none of which you control. The honest framing is that the edit improves your odds; the way to find out is to re-ask the same question after the engine has re-read the page and compare, treating any single comparison as one directional sample.
Look in your server or CDN access logs for the published AI crawler user agents — GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, Google-Extended for Google’s AI training access. Filter by the exact URL, not the domain, because whole-site totals hide the fact that your key page is never fetched. Most hosting platforms and CDNs expose this without extra tooling; if yours does not, an edge log drain or an analytics rule keyed on user agent will.
Start with the pages that map to questions you already track and already lose. Those are the ones where the mismatch between “we cover this” and “we are never credited for it” is provable rather than assumed. Commercial comparison pages and definitional pages tend to surface first, because they are the shapes assistants are most often asked to summarise, and they are also where buried answers and collapsed summaries are most common.
Lead with a short, direct answer to the exact question in the first screenful, before any narrative. Convert any substance that lives only in an image, chart or table into text. Expand anything hidden behind a click, since content the crawler cannot reach without interaction may as well not exist. Add verifiable specifics — figures, dates, named constraints — because assistants favour passages they can attribute safely. FAQPage schema helps a parser find the Q&A block once the answer is actually there.
You need two joined datasets: AI crawler hits per URL from your logs, and per-question citation records from a tracker. Neither on its own identifies the class. At portfolio scale the join is worth automating — export crawler hits grouped by URL, export the credited URLs from your tracked answers, and look for URLs present in the first set and absent from the second for questions they clearly address.
Not in the same way. If robots.txt or your CDN blocks GPTBot, ClaudeBot or their peers, the diagnosis is never-crawled by design, and no page rewrite will change it. That can be a deliberate, legitimate choice — some publishers block training access on purpose. The important thing is to know which situation you are in before spending on content, since the two look identical from a citation report alone.
Often enough to notice a regression, which in practice means on the same cadence as your citation tracking rather than continuously. Crawler behaviour and answer composition both shift week to week, so a page that is credited today can quietly stop being credited after a competitor publishes or an engine updates. A recurring check on a stable question set is more informative than an intensive one-off audit.
See which of your pages get read and passed over.
SearchChamp’s AI Visibility Tracker records who each AI answer actually credits for the questions you track — so an uncited page stops being a guess. 7-day free trial, cancel anytime.