Key takeaways
- Being indexed in Google only proves Googlebot found your page. It says nothing about whether GPTBot, ClaudeBot or PerplexityBot can reach it, or whether an answer engine would pick it as a source.
- Diagnose in order: crawler access, then alternate indexes like Bing, then content type and extractability, then freshness, then off-site presence. Fixing step 4 while step 1 is broken wastes time.
- Promptwatch Data shows product pages and landing pages make up about 39% of ChatGPT citations (Sep 28 to Oct 4, 2026). Page type matters as much as quality.
- Track results with crawler logs and a prompt tracker. Guessing from Google rankings will mislead you.
Why indexed and cited are different things
I see this complaint constantly: "We rank on page one. ChatGPT never mentions us." It feels like a bug. It isn't. Google ranking and AI citation are separate selection processes with different inputs.
An Ahrefs analysis of 15,000 prompts from September 2025 found that only about 12% of URLs cited by ChatGPT, Gemini and Copilot rank in Google's top 10 for the same prompt. Google AI Overviews lean on the organic top 10 more (about 38% of the time, per Ahrefs), but they are the exception. I haven't re-verified those figures against our own data, so treat them as directional.
So "indexed" is the entry ticket to one system. To get cited you need to be reachable by a different set of bots, retrievable through different search backends, and shaped like something an answer can quote.
The Netsujo diagnostic splits "not showing up" into three problems: not indexed, indexed but not ranking, and not appearing in AI. You are in the third bucket, but it is still worth a two-minute check that bucket one is really clean.

Step 0: confirm your Google indexing is really healthy
"Indexed" is often assumed rather than checked. Before blaming AI, verify:
- Search
site:yourdomain.comand thensite:yourdomain.com/the-pagefor the specific pages you want cited. - Open the Pages report in Google Search Console and look at the status of your key URLs. "Discovered, currently not indexed" and "Crawled, currently not indexed" are not the same as indexed.
- Check for stray noindex tags and X-Robots-Tag headers, a common leftover from staging.
- Check canonicals. A page that canonicalizes to another URL is not the page being indexed.
If only your homepage is indexed and the deeper pages you want cited are not, fix that first. Onely's advice for the "discovered, not indexed" status is to check internal links, sitemap quality, duplicate URLs, crawl waste and server performance before requesting indexing again.
Step 1: check whether AI crawlers can reach you
This is the most common failure and the cheapest to fix. Many CDNs, security plugins and WAF rules block AI bots by default, and your Google rankings will not warn you.
Read your robots.txt
Open yoursite.com/robots.txt and look for these user agents:
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Claude-User
PerplexityBot
Google-Extended
A Disallow: / next to any of them blocks that provider. A permissive version looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
There is a real distinction here. Training crawlers and answer-time fetchers are separate bots. You can block training while still allowing the bots that fetch pages when a user asks a question. The DEV Community audit makes the same point about GPTBot versus CCBot. Decide deliberately rather than inheriting a blanket block.

Look beyond robots.txt
robots.txt is only one layer. Bot protection in Cloudflare, Akamai or a WordPress security plugin can return 403s or challenge pages to AI user agents even when robots.txt says yes. JavaScript-only rendering is another quiet killer: if the content appears only after client-side scripts run, many AI fetchers will see an empty shell. Prerendering solutions such as Prerender.io exist for exactly this case.

Compare your logs to the market mix
Promptwatch Data's AI crawler traffic report (IP-verified requests, Jun 8 to Sep 6, 2026) found OpenAI accounted for 79.8% of verified AI crawler requests in the week of Aug 31 to Sep 6, down from 94.8% in early June. Its advice is a good diagnostic: if a provider holds a big share in the report but is near zero in your own logs, you are probably blocked or unreachable for it.
To do this you need server or CDN logs. Screaming Frog crawling with different user agents is a quick manual test, and log analyzers like Oncrawl and Botify can slice bot traffic. Platforms such as Promptwatch include IP-verified AI crawler logs, which matter because spoofed user agents otherwise pollute the picture.


Step 2: check Bing, not just Google
This one surprises people. ChatGPT Search has leaned on Bing's index for retrieval, so a site missing from Bing can be hard for ChatGPT to find regardless of Google rank. The fix is cheap:
- Search
site:yourdomain.comon Bing. - If results are thin, set up Bing Webmaster Tools.
- Submit your sitemap and request indexing for key pages.
- Consider IndexNow so new and updated URLs are pushed automatically.
Tools like IndexerNow automate IndexNow pushes. Do not treat third-party "instant indexing" services as a magic fix, though. If the page is thin or blocked, forcing a crawl just gets it rejected faster.
Step 3: look at what type of page you are offering
Here is where I would spend the most thought, because it is the least obvious. AI engines do not cite all content types equally.
Promptwatch Data's ChatGPT citation share by content type (Sep 28 to Oct 4, 2026) breaks it down:
| Content type | Share of ChatGPT citations |
|---|---|
| Product pages | 22.4% |
| Landing pages | 16.8% |
| Listicles | 13.6% |
| How-to guides | 8.7% |
| Documentation | 7.5% |
| General articles | 6.1% |
| Comparisons | 5.9% |
| News | 5.7% |
| FAQ pages | 3.3% |
| Reviews | 1.7% |
The split shifts by prompt intent. For commercial prompts, product pages (26.0%), landing pages (20.6%), listicles (15.0%) and comparisons (7.9%) make up almost 70% of citations. For informational prompts, how-tos (10.7%) and docs (8.5%) climb. For branded prompts, product and landing pages dominate at 23.3% and 22.8%.
The practical read: if your only indexed assets are opinion blog posts, and your prompts are commercial, you are competing in a format that is a minority of citations. Ask whether you have a clear product or landing page that states what you sell, for whom, with specs and pricing. Ask whether any "best X" roundup includes you.
Step 4: test extractability
An AI engine builds an answer from fragments. If your page cannot supply a clean fragment, it loses to one that can. Open your key page and ask: can I lift a specific, quotable answer out of the first 300 words?
Low extractability looks like this:
We are a leading provider of innovative solutions that help
businesses transform their digital presence.
High extractability looks like this:
Our deployment platform cuts CI/CD pipeline time from 12 minutes
to 3.2 minutes. 4,200 teams use it to ship to production.
The second version has numbers, a subject and a claim. The first could describe a thousand companies. A quick edit pass:
- Put the direct answer under each question-style heading, in the first sentence.
- Use concrete figures, names and dates instead of adjectives.
- Add FAQ sections where people really do ask those questions.
- Use tables for comparisons and specs. They are easy to parse.
- Add structured data (Organization, Product, FAQPage, Article) where it truthfully applies. Schema App is one platform for managing this at scale; Yoast SEO and AIOSEO cover the basics on WordPress.
One caution: do not chase formats for their own sake. Promptwatch Data's markdown report found markdown files are just 0.05% of AI search citations. Markdown serves AI agents, not AI search, so don't spend a week publishing .md copies of your pages hoping for citations.
Step 5: audit freshness by content type
The common advice is "AI engines love fresh content." It is half true, and the data says the answer depends on what you published.
Promptwatch Data's content half-life study followed every page ChatGPT cited on July 6, 2026 for 90 days:
| Content type | Still cited after 30 days | Still cited after 60 days |
|---|---|---|
| Landing pages | 78% | 69% |
| Docs | 76% | 67% |
| Comparisons | 72% | 63% |
| Listicles | not reported | 60% |
| Product pages | not reported | 55% |
| News | not reported | 41% |
News loses half its cited pages after about 36 days. Product pages, reviews and how-tos reach half-life around 74 to 77 days. Landing pages, docs, comparisons and listicles last long enough that their half-life is likely near or beyond 90 days.
So what does that mean for your checklist? If your content is mostly news-style posts, expect churn and plan to keep publishing. If your content is evergreen docs and landing pages, staleness is less likely to be the cause of your absence, and you should look at steps 1 to 4 first. If a page was once cited and then dropped, a refresh with updated facts, dates and pricing is worth trying.
Step 6: check off-site presence
AI answers rarely rely on one domain. Promptwatch Data's unique domains report found almost half of ChatGPT responses cite 10 or more unique domains (Sep 28 to Oct 4, 2026). You are competing for one slot among many, and a lot of those slots go to third parties.
Things to look at:
- Are you included in the roundups and comparison pages that AI cites for your category? Listicles are 13.6% of ChatGPT citations, so being named in someone else's "best of" list matters.
- Do trusted review sites mention you? Reviews are a small citation share, but they feed branded prompts.
- Is your documentation public and crawlable?
- Is your brand named in third-party pages even without a link?
Also note that domain authority is not a gate. Promptwatch Data's domain rank report shows sites with domain rank 46 to 75 earned almost half of ChatGPT citations in August 2026, while DR 91 to 100 fell to about 3%. A mid-sized site can get cited. Social platforms are also uneven: ChatGPT drew just 0.46% of citations from social platforms versus 12.2% for Google AI Overviews (Sep 28 to Oct 4, 2026, per the social citations report), so the right off-site play depends on which engine you care about.
Step 7: measure, don't guess
Once you have made changes, you need evidence. A fix that is not measured is a hope. The checks that tell you the most:
- Crawler logs: did the bots you unblocked actually start visiting?
- Prompt tracking: for a fixed set of prompts, are you mentioned and cited, and where in the source list?
- Referral traffic: does AI-sourced traffic show up in analytics?
A few ways to cover this:
| Need | Option | Notes |
|---|---|---|
| Quick free snapshot | AnswerGist, HubSpot AI Search Grader | One-off diagnostics |
| Prompt monitoring only | Otterly.AI, Peec AI | Tells you if you were mentioned, not why |
| Monitoring plus crawler logs, citations and content actions | Promptwatch | Tracks real AI interfaces, logs bot visits, and ties content gaps to fixes |
| Enterprise suites | Profound, Botify | Heavier and priced accordingly |


The reason I lean toward tools that include crawler logs is the diagnostic itself. A pure prompt tracker can say "you were not cited." It cannot say whether the bot ever fetched the page, hit an error, or read it and passed. For a deeper list of options, the GEO software directory covers most of the field.
The checklist in one place
- Confirm real Google indexing for the specific URLs (not just the homepage).
- Check robots.txt for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended.
- Check CDN, WAF and security plugin rules for 403s or challenges against AI user agents.
- Confirm server-rendered content, not an empty JavaScript shell.
- Verify Bing indexing, submit a sitemap, consider IndexNow.
- Compare your crawler logs to the market mix. Large provider, near-zero visits means blocked.
- Match page types to prompt intent: product, landing, listicle, comparison, docs, FAQ.
- Rewrite key pages so a direct, quotable answer sits in the first 300 words.
- Add truthful structured data.
- Refresh pages whose content type decays quickly, especially news.
- Get included in third-party roundups, reviews and public docs.
- Re-measure with logs and a fixed prompt set after each change.
What I would do first on Monday
If you only have an hour: do the robots.txt and WAF check, then the Bing check. Those two account for a disproportionate share of "indexed but invisible" cases, and both are fixable the same day. Then pick your three most important commercial pages and rewrite their opening paragraphs with specific facts. If you are working with an SEO partner, ask them to show you bot-level log data rather than ranking screenshots. Ranking screenshots will keep telling you everything is fine.
Google's own guidance, incidentally, says it rewards high-quality content however it is produced. That is a statement about Search, not about ChatGPT or Perplexity. For those, you have to earn the citation on their terms.
