Why Isn't My Site Cited by AI Search? A Diagnostic Checklist

By Citable Press · 2026-09-07

If your site isn’t showing up in ChatGPT, Perplexity or Google’s AI Overviews, the cause is almost always one of four things: AI crawlers can’t reach your pages, they can’t read what’s rendered, your content isn’t structured for extraction, or nobody outside your own site is talking about you. Each of those has been measured, separately, with real sample sizes — and the fixes are different for each one. This piece walks through the diagnosis in the order that actually matters, starting with the failure that silently overrides everything else.

Key Takeaways

• A July 2026 crawl of 10,894 domains found only 15% blocked at least one AI crawler outright, but 39.5% of GPTBot bans were being ignored by the very sites that set them (HasData, 2026).
• Most AI crawlers do not execute JavaScript at all; a joint Vercel and MERJ analysis tracking over 500 million GPTBot fetches found zero evidence of JS execution (Vercel & MERJ, via Averi, 2026).
• Ahrefs’s controlled test of 1,885 pages found adding schema markup produced no meaningful citation lift on ChatGPT or AI Mode, and a small decline on AI Overviews (Ahrefs, 2026).
• A 2026 measurement framework covering 21,143 AI citations found the pages that get quoted most heavily are longer, more structured, and richer in extractable facts — not just present (arXiv:2604.25707, 2026).
• Hard paywalls appear to be close to fatal: a July 2026 test of 40 queries on Claude found paywalled outlets like the Wall Street Journal received 0% of citations (5W Public Relations, via AuthorityTech, 2026).

Can AI Crawlers Even Reach Your Site?

This is the check most guides skip, because it’s boring and technical. It is also the one that makes every other fix pointless if it fails.

Start with robots.txt. A July 2026 crawl of 10,894 domains — the Tranco top 10,000 plus 1,148 news publishers — found that only 15% of the full sample blocked at least one AI crawler (HasData, 2026). Publishers were the exception: they blocked far more often than the open web. But the same study found something stranger. Among domains that Google’s AI Mode actually cited, 51.9% blocked at least one AI crawler in robots.txt — more than three times the baseline rate (HasData, 2026). Blocking GPTBot, in other words, does not reliably keep you out of Google’s AI answers.

Enforcement is inconsistent too. The same HasData research found 39.5% of GPTBot disallow rules were being ignored by the sites that set them, and when researchers requested the same page as a browser versus as GPTBot from an identical datacentre IP, the browser request succeeded 83.8% of the time against GPTBot’s 54.2% (HasData, 2026). Your written policy and your server’s actual behaviour can quietly disagree.

Hosting matters more than most site owners realise. A September 2026 check of 9,037 live sites found Cloudflare-hosted domains blocked GPTBot at 22.4%, versus 5.2% on Vercel, 4.0% on Netlify, 3.9% on Fastly and 3.4% on CloudFront (directree, 2026). Cloudflare’s own default-block policy, which took effect on 15 September 2026, covers a further 8.5% of the top web (HasData, 2026). If you migrated hosting or enabled a CDN feature this year, check whether it silently changed your AI crawler policy.

A separate quarterly crawl of 1,744 sites found 9.9% blocked GPTBot and 84.2% had no AI crawler policy at all, most of them not by deliberate choice (AI Visibility, 2026). For example, an inherited robots.txt from a theme or starter template can quietly disallow crawlers nobody on the team has ever reviewed.

Is Your Content Hiding Behind JavaScript?

If the crawler reaches your page but the content only appears after a script runs, the crawler sees an empty shell.

A joint analysis by Vercel and MERJ tracked over 500 million real GPTBot fetches and found zero evidence of JavaScript execution (Vercel & MERJ, via Averi, 2026). GPTBot fetches JavaScript files in roughly 11.5% of requests but never runs them, and ClaudeBot fetches JS in about 23.84% of requests with the same result (SearchOptimo, 2026). Googlebot is the outlier: it uses a headless Chrome rendering pipeline and can eventually process client-side content, which is why the same page can rank in Google while remaining invisible to ChatGPT or Perplexity (SearchOptimo, 2026).

Consider a single-page application that loads its product descriptions, FAQs or pricing tables through client-side JavaScript. A human sees a complete page. An AI crawler sees raw HTML with none of that content in it. The fix is server-side rendering or prerendering for anything you want quoted — not a JavaScript optimisation, but a decision about what ships in the first HTML response.

Is Your Content Built So AI Can Extract It?

Reachable and readable are necessary conditions, not sufficient ones. The next question is whether an AI system can lift a clean, standalone claim from your page without needing the rest of the article for context.

A 2026 measurement framework analysed 602 controlled prompts across ChatGPT, Google AI Overview and Perplexity, covering 21,143 valid citations and 18,151 successfully fetched pages (arXiv:2604.25707, 2026). The pages that shaped the generated answer most heavily, rather than just appearing as a footnote, tended to be longer, more modular, and dense with definitions, numbers, comparisons and step-by-step instructions (arXiv:2604.25707, 2026). The same paper carried an important caveat forward: simply reformatting content into Q&A blocks did not, on its own, improve how deeply a page was absorbed into an answer (arXiv:2604.25707, 2026). Structure has to carry real evidence, not just headings that look like questions.

This lines up with what a separate industry study found when it isolated ranking position from AI citation. An analysis of 863,000 keywords and 4 million AI Overview URLs found only 38% of citations now come from pages ranking in Google’s top 10, down sharply from 76% in mid-2025 (Ahrefs, via DigitalApplied, 2026). Pages ranked 11–100 accounted for 31.2% of citations, and pages beyond rank 100 picked up another 31.0% (Ahrefs, via DigitalApplied, 2026). Moz found a similar split from a different angle: across nearly 40,000 queries, 88% of Google AI Mode citations came from outside the organic top 10 (Moz, via AuthorityTech, 2026). If your page ranks well but never gets quoted, ranking was never the blocker — extractability was.

Paywalls sit at the extreme end of this problem, because a system that cannot read the content cannot extract anything from it. A July 2026 test of 40 queries across six categories on Claude found hard-paywalled outlets — the Wall Street Journal, the Financial Times, Bloomberg — received 0% of citations, and metered sites like the New York Times fared no better (5W Public Relations, via AuthorityTech, 2026). Open-web publishers captured 91.3% of citations in the same test (5W Public Relations, via AuthorityTech, 2026). A related Rutgers/Wharton analysis found major publishers that block LLM crawlers outright lose roughly 23% of their weekly traffic (Rutgers/Wharton, via AuthorityTech, 2026). For a deeper look at what separates cited pages from ignored ones across a much larger sample, see our hub post on what gets cited by AI search.

Does Anyone Else Vouch for You?

Access and structure explain whether a page can be cited. They don’t explain whether it’s trusted enough to be selected in the first place. That’s a separate, off-site question.

Muck Rack’s analysis of 25 million cited links across ChatGPT, Claude and Gemini found 84% of all AI citations traced back to earned media — third-party coverage, not brand-owned content (Muck Rack, via Machine Relations, 2026). Reviews carry similar weight: a Trustpilot analysis of over 800,000 AI responses found brands with no active review profile were cited in just 1% of answers, while brands that actively collected and responded to reviews were cited in 75.3% (Trustpilot, via AuthorityTech, 2026).

The order of diagnosis matters more than any single fix on this list. Most advice starts with content quality or schema markup, because those are the levers a writer controls directly. But an access failure — a blocked crawler, an unrendered page, a hard paywall — zeroes out every downstream improvement before it has a chance to matter. Fix reachability first, structure second, and off-site authority third; doing them in reverse order means polishing content that a crawler never sees.

We ran our own small check of this off-site pattern on 2 September 2026, querying Google, Perplexity and ChatGPT with the identical prompt “best GEO / AI-visibility tools” on the same day and logging every domain each one cited. Across the three engines, they cited roughly 35 distinct domains between them, and only one domain was cited by all three. That’s a single, same-day snapshot rather than a large study, but it matches the wider pattern above: each engine draws on a different slice of the earned-authority web, so being cited by one does not mean you’ll be cited by another.

Do Schema Markup and llms.txt Deserve the Effort?

Both are common recommendations, and both have now been tested directly rather than just correlated with citation counts.

Ahrefs’s first pass looked at 6 million URLs and found AI-cited pages were roughly three times more likely to carry JSON-LD schema than uncited pages — the correlation most “add schema” advice is built on (Ahrefs, 2026). To test causation, the same team tracked 1,885 pages that added schema between August 2025 and March 2026, matched against 4,000 control pages with similar pre-existing citation levels. The result: AI Mode and ChatGPT citations moved by amounts statistically indistinguishable from noise, and AI Overviews citations fell by a small but statistically significant margin (Ahrefs, 2026). Ahrefs’ own conclusion was blunt: “if you’re already doing the rest of the SEO work well, JSON-LD isn’t going to be the unlock” (Ahrefs, 2026).

llms.txt fares no better under testing. SE Ranking analysed roughly 300,000 domains using both correlation tests and a predictive model, and found no measurable relationship between having an llms.txt file and citation frequency (SE Ranking, 2026). Adoption sat at just 10.13% of the sample, and removing the llms.txt variable from their model actually improved its predictive accuracy (SE Ranking, 2026). A separate July 2026 crawl put adoption even lower, at 7.4% of 10,894 domains (HasData, 2026). Neither is harmful to add. Neither should be mistaken for a fix.

Key Terms Worth Knowing

Crawlability is whether an AI crawler’s HTTP request to your page succeeds and returns readable HTML, unaffected by robots.txt rules, firewall challenges, or login walls.

Citation absorption is the degree to which a cited page’s language, facts or structure actually shape an AI-generated answer, as distinct from merely being listed as a source (arXiv:2604.25707, 2026).

Entity clarity is how unambiguously an AI system can identify who or what your page is about, typically strengthened by consistent naming, third-party corroboration, and knowledge-graph presence rather than by on-page markup alone.

llms.txt is a proposed plain-text file, similar in spirit to robots.txt, that summarises a site’s content for AI systems — currently adopted by roughly 7–10% of sites and with no demonstrated effect on citation rates (SE Ranking, 2026).

The Diagnostic Checklist

Use this in order. Each row assumes the ones above it have already been ruled out.

For a broader read on how each AI crawler differs technically, see the comparison below.

Why any of this matters at scale: Pew Research Center’s survey of 5,119 US adults, fielded 17–23 February 2026, found 49% now use AI chatbots and 60% read AI-generated summaries in search results (Pew Research Center, 2026). Being unreachable to AI crawlers is no longer a niche technical footnote.

How We Checked This

Every statistic in this piece is attributed to a named source with a publication date, and a sample size wherever the original source provided one. Where a study qualified its own finding — Ahrefs on the difference between correlation and causation, or the arXiv paper’s own note that Q&A formatting alone didn’t help — we carried that qualification through rather than smoothing it out. We did not test every claim ourselves; where we ran our own check, it’s marked and described as exactly what it was: a same-day, three-engine snapshot, not a large study. Our full approach to sourcing and corrections is on our methodology page.

The Bottom Line

Most sites that aren’t cited by AI fail at the access stage, not the content stage: a blocked crawler, an unrendered page, or a paywall, none of which any amount of schema markup or llms.txt can fix. Check reachability first — robots.txt, hosting, firewalls — then whether your content survives without JavaScript, then whether it’s structured with genuine extractable evidence rather than just headings. Only after those three are solid does off-site authority, the reviews, the press, the forum mentions, become the binding constraint. For the fuller picture of what separates cited pages from ignored ones across a much larger sample, our hub post on AI citations is the next place to look.