Key takeaways
- Meta AI, Claude Code and Codex are three very different discovery surfaces. One is a consumer assistant, two are coding agents. Tracking them in one place only works if you measure each for what it actually does.
- Crawler logs are the cheapest early signal. Promptwatch Data shows Meta-WebIndexer going from about 2.2% to 37.8% of tracked AI crawler requests between mid-July and August 9, 2026.
- For software prompts, docs and help centers earn the biggest share of ChatGPT citations. Treat docs as your main retrieval surface for every assistant.
- Public facts about how Claude Code and Codex retrieve information are thin. Test them yourself with a fixed prompt set rather than trusting vendor claims.
- A good setup combines a prompt monitor, crawler logs and referral analytics, and then feeds gaps back into docs and content.
Why these three surfaces belong in one report
A developer who used to Google "best vector database for RAG" and open eight tabs now types a sentence into a terminal. The agent picks a library, writes the import line and moves on. Meanwhile the same person's non-technical colleague asks Meta AI inside WhatsApp or Instagram which tool to buy for the team.
That split is why tracking them together is useful. Your buyer and your user may hit different assistants, and your brand can be recommended in one and invisible in another.
Here is the honest caveat up front. Of the three, only Meta has public, measurable crawler behavior in the data I could verify. I found no independent figures on how Claude Code or Codex retrieve information, and I won't invent any. What follows is a framework built on what is measurable, plus a testing method for the rest.
What each surface does differently
| Surface | Who uses it | How it finds your product | What you can measure today |
|---|---|---|---|
| Meta AI | Consumers, founders, non-technical buyers | Reportedly moving to its own web index | Meta crawler hits in your logs, answers from prompt tracking |
| Claude Code | Developers working in a terminal or IDE | Model knowledge plus whatever web search, docs or MCP tools the session uses | Recommendations in test sessions, Anthropic crawler hits |
| Codex | Developers using OpenAI's coding agent | Model knowledge plus web search when enabled | Recommendations in test sessions, OpenAI crawler hits |
The common thread is that all three answer from a mix of what the model already knows and what it fetches at answer time. You influence the first slowly and the second quickly.
Meta AI: the surface that is changing fastest
Promptwatch Data tracked Meta-WebIndexer climbing from about 2.2% of AI crawler requests in mid-July 2026 to 37.8% on August 9, a roughly 17x jump in under a month. By that day it was the heaviest AI crawler in the logs, at more than one request in three. The Meta crawler report notes the surge comes from the indexing crawler specifically, not Meta's training or link-fetching bots.
Why does it happen? There are reports that Meta is building its own web search index so Meta AI doesn't depend on Google or Bing. The source is a post from @levelsio who said Meta staff told him this "allegedly", after Meta's scraping triggered load alerts on his server. Treat it as unconfirmed. The crawl data itself is not in doubt, though.
What to do about it:
- Check your server logs and robots.txt for Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher and FacebookBot.
- Decide on purpose whether to allow them. If you block them now, you may be out of Meta's index when it matters.
- If your infrastructure bills per request, budget for the load.
One measurement note: that 37.8% is Meta-WebIndexer requests divided by all tracked AI crawler requests. It excludes Googlebot, Bingbot and human visitors, so it is not a share of your total traffic.
Claude Code and Codex: test, don't assume
Coding agents are hard to measure because the answer depends on the session. A fresh project, an installed MCP server, a web search toggle and the model version can all change what gets recommended.
Vendors in this space argue that the funnel has collapsed from six steps to two or three, and that agents pick defaults from retrieved material like llms.txt files, Agent Skills and MCP servers. Cite Solutions lays out this stack in a five-layer model, from docs structure up to third-party retrieval pools. It is a useful mental model. It is also vendor marketing, so don't read it as proof of how Claude Code or Codex behave internally.
A practical way to test without trusting anyone:
- Write 20 to 40 realistic developer prompts, such as "scaffold a SaaS with auth, Postgres and deploy on Vercel" or "add background jobs to a Node app".
- Run each in Claude Code and Codex on clean projects, with web search on and then off.
- Record which packages, services and docs get named, and in what position.
- Repeat monthly, since models and defaults change.
- Note which sources the agent fetched, where the tool shows them.
This is manual, and it is slow. Still, it is the only honest way to learn how your product shows up at the point where the import line gets written. Promptwatch also lists Codex, Claude Code and OpenCode among the coding agents it tracks, which can automate part of the loop.
Use crawler logs as your common denominator
Prompt tracking tells you what assistants say. Crawler logs tell you whether they can read you at all. That makes logs the one signal that works across all three surfaces.
Promptwatch Data reports IP-verified crawler requests from June 8 to September 6, 2026. OpenAI accounted for 79.8% of verified AI crawler requests in the week of August 31 to September 6, down from 94.8% in the week of June 8 to 14. The mix is shifting, and Meta is a large part of why.
The agents you care about map to these user agents:
| Provider | Crawlers worth watching |
|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot |
| Meta | meta-webindexer, meta-externalagent, meta-externalfetcher, FacebookBot |
| Google-Extended, Google-Agent | |
| Perplexity | PerplexityBot, Perplexity-User |
Two checks pay off quickly. First, compare your own logs with the public mix. A provider that is big in the benchmark but near zero on your site probably means a CDN or WAF rule is blocking it. Second, separate training crawlers from answer-time fetchers. Only the second group tells you someone asked a question and your page was pulled in to answer it.
Promptwatch handles this with IP-verified crawler logs that filter out spoofed bots, and it works with Cloudflare, Vercel, Fastly and raw server logs.
Make your docs the retrieval surface
For software, docs are the content assistants lean on. Promptwatch's ChatGPT software citation data covers organic prompts from September 28 to October 4, 2026:
| Source type | Share of software citations |
|---|---|
| Docs and help centers | 2.64% |
| Vendor sites | 2.01% |
| Review sites and analysts | 1.57% |
| GitHub | 1.2% |
| App stores | 0.47% |
The top domains were github.com at 1.2%, learn.microsoft.com at 0.98% and g2.com at 0.73%. Vendor sites plus docs earned 4.65%, about three times what review sites and analysts did. The report's own takeaways are to publish crawlable docs and help pages, keep a public GitHub presence for dev tools, and keep G2 and Capterra profiles current.
Two warnings. This is ChatGPT data only. It says nothing directly about Meta AI, Claude Code or Codex. And the shares look small because they are spread across a huge number of domains. Read them as relative weights, not as a ceiling.
Concrete doc fixes that help every assistant:
- Render docs as plain HTML that a crawler can read without running JavaScript.
- Give each page one clear job, with the product name and the task in the heading.
- Keep quickstarts short and copy-pasteable, since agents lift code almost verbatim.
- Publish an llms.txt if you like, but don't count on it. Markdown files are only 0.05% of AI search citations, a format that serves agents more than AI search.
- Keep pricing and comparison pages factual. Those are the pages that get cited for buying questions.
Choosing tools for the job
The market is crowded. You need coverage of consumer assistants like Meta AI, coding agents, and a way to see crawler activity. Most trackers cover ChatGPT, Perplexity, Gemini and Claude, but few cover all three of our surfaces. Check each vendor's current coverage before you buy, as I could not verify pricing or features for this guide.
| Need | Option | Note |
|---|---|---|
| Meta AI, coding agents and crawler logs in one platform | Promptwatch | Tracks Meta AI, Codex, Claude Code and OpenCode, plus IP-verified crawler logs |
| Enterprise monitoring | Profound | Strong monitoring, confirm surface coverage |
| Light prompt tracking | Otterly.AI, Peec AI | Monitoring focused, no crawler logs |
| Budget SaaS tracking | Trakkr.ai, LLM Pulse | Check engine lists first |
If you are also assembling a shortlist, the GEO software directory is a good place to compare options by feature.
On the SaaS marketing side, guides like Sight AI's implementation walkthrough start with a sensible baseline audit. Run the same customer-style prompts across platforms, log whether you were mentioned, note the sentiment, and watch which competitors appear where you should. That method carries over to coding agents with prompts rewritten as developer tasks.
A simple monthly workflow
- Build two prompt sets. One for buyers ("best X for Y team") and one for developers ("set up X in a Next.js app").
- Run the buyer set across Meta AI and other consumer assistants. Run the developer set in Claude Code and Codex.
- Pull crawler logs and check for each provider listed above. Fix any blocks.
- Track AI referral traffic and conversions, not just mentions. Mentions without visits are a vanity number.
- Turn gaps into work: a missing doc page, a weak comparison page, an outdated G2 profile.
- Re-run and compare. Expect noise. Single runs of an AI answer vary, so look at trends over weeks.
What to read into the results
- You are mentioned but never cited: your content is probably not crawlable or not specific enough.
- You appear in Meta AI but not in coding agents: your consumer brand is stronger than your developer docs.
- You appear in coding agents with the wrong version or deprecated API: refresh your docs and quickstarts, and publish a clear changelog.
- Crawlers hit your site but you get no mentions: the content is being read and not used. Look at page structure and how directly you answer the question.
Where this leaves you
The honest state of play in October 2026 is lopsided. Meta's crawler growth is well documented. Claude Code and Codex retrieval is mostly a black box, so a repeatable manual test beats any claim you read. Start with logs and docs, add a prompt set for each surface, and let your own data tell you where the gaps are.
For wider reading on tracker options, the AI rank tracking directory lists many of them.