Key takeaways
- DataDome counted 17.7 billion AI agent requests in Q2 2026, up 45% from 12.2 billion in Q1. In June, Meta-WebIndexer passed Meta-ExternalAgent in monthly volume for the first time.
- Crawl volume and referral value point in opposite directions. ChatGPT-User requests fell 6% quarter over quarter at DataDome, yet ChatGPT still accounts for 80-88% of AI referrals.
- Not all AI bots do the same job. Training crawlers, search indexers and user-triggered fetchers need different decisions in robots.txt and your CDN.
- Most of these bots fetch HTML and never run JavaScript, so your analytics tag will not see them. Server or CDN logs are the only reliable source.
- Individual bot numbers swing wildly. Treat any single week of data as noise and look at trends per crawler, not totals.
What actually changed in your logs
If you looked at your server logs in January and again this autumn, you probably noticed the user-agent list got stranger. Meta crawlers that barely registered a year ago now sit near the top. ChatGPT-User, once the headline agent, bounces around. Bots you never heard of show up with tens of thousands of hits.
The numbers back up the feeling. According to DataDome's Q2 2026 AI traffic report, its network processed 17.7 billion AI agent requests from April through June, a 45% jump from Q1. Monthly volume went from 4.77 billion in April to 6.29 billion in May and 6.60 billion in June. That is one vendor's network, so treat it as a directional signal rather than a census of the web.
A separate view comes from Cloudflare Radar. As summarised by MarketSplash, all bots together (search engines, scrapers, attack scripts and AI agents) made 39.2% of HTTP requests from October 11 to November 8, 2026. AI is a slice of that, not the whole thing. When you read a claim that bots now outnumber people, check which bots the number includes.
Meta is now the loudest crawler
The biggest story in the Q2 data is Meta. DataDome reports Meta-ExternalAgent up 74% quarter over quarter and Meta-WebIndexer up 163%. In June, WebIndexer overtook ExternalAgent in monthly volume.
Promptwatch's own crawler logs show the same shape, only sharper. In Promptwatch Data's Meta-WebIndexer report, the crawler sat near 2.2% of all tracked AI crawler requests in mid-July 2026, spiked to roughly 23% around July 20 to 22, then climbed to 37.8% on August 9. That is about a 17x increase in under a month, and it made Meta-WebIndexer the heaviest AI crawler in the logs for that period.
Two caveats on that comparison. DataDome measures requests on its protected network. Promptwatch measures share of verified AI crawler requests in its logs, with Meta's other bots counted only in the denominator. Different denominators, different crawler sets. Cite them separately and don't try to reconcile the percentages.
Why is Meta doing this? On August 6, the developer @levelsio posted that Meta staff had allegedly told him the company is building its own web index, after the crawl set off load alerts on his server. That is unconfirmed, and I would hold it loosely. What is confirmed is the timing: BrightEdge recorded a 30-fold jump in Meta-identified requests on one ecommerce client on September 7, from about 30,000 a day to about 1.0 million, and Meta announced its Muse personal agent the next day. Whatever the plan is, the crawl came first.

The referral side is much less impressive. MarketSplash, citing DataDome, reports that Meta's two crawlers made 12.4 billion requests in Q3 2026, about half of the AI agent traffic DataDome recorded, against 51,000 referrals from Meta AI. Huge crawl, almost no traffic back. That ratio matters if your hosting bill scales with requests.
ChatGPT-User: smaller crawl, bigger payoff
ChatGPT-User is the fetcher that opens a page while ChatGPT is answering a person's question. A request from it is closer to a citation event than to indexing. One TechSEO thread analysing 48 days of server logs put it bluntly: each request represents a real person's conversation touching your URL.
In DataDome's data, the top agent in Q1 was ChatGPT-User, and it declined 6% in absolute volume in Q2. Meanwhile ChatGPT took 80-88% of all AI referrals every month and grew 17% quarter over quarter. So the bot that fetches for people is fetching a bit less overall while the referrals it feeds keep growing. Volume in your logs and visits in your analytics are different currencies.
This bot is also volatile on individual sites. MarketSplash reports that ChatGPT-User's requests to evilmartians.com fell from 196,973 between May and July 2026 to 16,445 in September and October, a 92% drop, with nothing changed on the site. If you see a cliff like that, check for an upstream change at the provider before you blame your own content.
There is also a behavioural change worth knowing about. Promptwatch's site: operator report shows ChatGPT Search started using the site: operator at scale on August 8, 2026, jumping from about 0.4% to about 17% of fanout queries overnight, with searches per response nearly doubling. Shifts like that change what the live fetchers go and read.
Who is who: a field guide to the user agents
The bots fall into three jobs. Mixing them up leads to bad robots.txt decisions.
- Training crawlers collect pages that may feed model training. Example: GPTBot.
- Search indexers build the index an AI search feature draws on. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and probably Meta-WebIndexer.
- User-triggered fetchers open one page because a person asked. Examples: ChatGPT-User, Claude-User, Perplexity-User.
OpenAI says robots.txt rules may not apply to ChatGPT-User fetches, since a person initiated them. Anthropic and Perplexity split their bots along the same lines. The practical upshot: blocking GPTBot does not block ChatGPT from reading a page a user pastes into a chat.
| Bot | Operator | Job | What its traffic tells you |
|---|---|---|---|
| GPTBot | OpenAI | Training collection | Whether you are in the training pool |
| OAI-SearchBot | OpenAI | Search indexing | Whether ChatGPT search can find you |
| ChatGPT-User | OpenAI | Live fetch for a person | Closest thing to a citation in the logs |
| ClaudeBot / Claude-User | Anthropic | Training / live fetch | Claude visibility and live reads |
| PerplexityBot / Perplexity-User | Perplexity | Indexing / live fetch | Perplexity answer access |
| Meta-WebIndexer | Meta | Index building (emerging) | Meta AI search exposure |
| Meta-ExternalAgent | Meta | Training and agent crawling | Crawl load, little referral value so far |
| Meta-ExternalFetcher | Meta | User-triggered fetch | Meta AI reading a specific page |
Promptwatch's AI crawler traffic report, covering June 8 to September 6, 2026, tracks IP-verified requests from OpenAI, Anthropic, Google, Perplexity and Mistral. OpenAI made up 79.8% of those requests in the week of August 31 to September 6, down from 94.8% in the week of June 8 to 14. The mix is diversifying, even if OpenAI still dominates. Note that Meta is not in that chart, so don't add its percentages to the Meta report's.
For a full directory of crawler names with robots.txt guidance, the AI bots and web crawlers directory lists 400+ of them.
Why your analytics will not show any of this
The bots that matter here mostly fetch HTML and leave. They do not execute JavaScript, so a tag-based analytics tool never fires. Vercel and MERJ tested the major AI crawlers, including GPTBot, ChatGPT-User, ClaudeBot and PerplexityBot, and found none of them rendering JavaScript.
Two consequences follow:
- A 30x jump in Meta requests can pass through your dashboards without moving a line, as BrightEdge pointed out.
- If your key content only appears after client-side rendering, these bots may see an empty shell. Server-side rendering or prerendering fixes that.
So the data source has to be raw logs: your web server, CDN or WAF. And the user-agent string alone is not trustworthy, since anyone can spoof it. Verify against the operator's published IP ranges. Promptwatch does this in its AI crawler logs, which is why spoofed bots drop out of its numbers.
How to count AI agents in your own logs
Here is a workable routine that takes an afternoon.
1. Pull a clean window
Export at least 30 days of raw access logs. One week is too short given how much individual bots swing. Include user agent, IP, path, status code and timestamp.
2. Group by operator and job
Use the table above. Report training, search indexing and live fetch separately. A grand total of AI requests hides the one number you care about, which is live fetches.
3. Isolate the fetchers on your important pages
Take your top commercial and informational pages. Count ChatGPT-User, Claude-User and Perplexity-User requests on them, next to human sessions on the same pages. That pairing is the figure worth putting in a report.
4. Check status codes
A crawler getting 403s or 429s from your WAF is a different problem from a crawler that reads you and never cites you. Promptwatch's own advice is blunt: if a provider is a large share in the benchmark but near zero in your logs, you are probably blocked or unreachable for that provider.
5. Compare against citations
Crawl without citation means a content problem. Citation without crawl usually means the model already knows you or fetched through a different path. Tying the two together is where the insight is.
What to allow, throttle or block
There is no universal answer, but a sensible default exists.
| Situation | Suggested approach |
|---|---|
| You want AI search visibility | Allow search indexers and user fetchers |
| You object to training use | Disallow training crawlers such as GPTBot, accept that live fetchers may still read pages |
| Infrastructure billed per request | Rate-limit Meta crawlers at the CDN, check crawl-delay support |
| Unknown bot with a big spike | Verify IPs first, then decide |
| Key pages rely on client-side JS | Add server rendering or prerendering before worrying about robots.txt |
My opinion: blocking Meta wholesale is a hasty call if you sell to consumers, because Meta AI and Muse could become a discovery surface. But it is completely reasonable to throttle it if the load hurts. Promptwatch recommends treating Meta as an emerging AI search surface and budgeting for sustained crawl load if you pay per request. That reads right to me.
If you need to generate rules or check what is currently reachable, the free Robots.txt generator for AI and the Agentic readiness check are quick starting points.
Tools for watching bot traffic
You have a few categories to choose from. Log analyzers and technical SEO crawlers cover the raw-log side, while AI visibility platforms link crawling to citations and traffic.
For tying bot activity to outcomes, Promptwatch pairs IP-verified crawler logs with citation tracking and AI visitor analytics, so you can see whether a page was read, cited and clicked. It also tracks Meta-WebIndexer and Meta's other crawlers alongside the OpenAI, Anthropic, Perplexity and Google bots.

For enterprise technical SEO with bot-level reporting, Botify is an established option.
FandangoSEO is a lighter website crawler and log analyzer.

BrightEdge, which published the Muse crawl data, tracks AI agent activity as part of its enterprise suite.

If you want to compare more options, the GEO software directory at bestgeosoftware.com lists dozens of platforms.
Referral traffic needs separate attention
Crawl data tells you who read you. It does not tell you who sent visitors. Keep a separate view for AI referrals, and note that ChatGPT stopped sending a referrer on October 8, 2026. Tagging links with utm_source=chatgpt.com and counting that source as ChatGPT is a stopgap many teams now use. Promptwatch's guide to attributing AI traffic walks through the options.
Citation behaviour also shifts under you. Promptwatch Data shows reddit.com's share of ChatGPT citations falling from about 4% to 0.5% on August 14, 2026, and that the content type most cited by ChatGPT is product pages at 22.4% (September 28 to October 4, 2026). If your fetcher traffic changes, those shifts are one place to look first.
A short checklist
- Keep 30+ days of raw logs and review them monthly.
- Verify bot IPs rather than trusting user agents.
- Report fetchers, indexers and trainers separately.
- Watch Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher and FacebookBot in robots.txt and WAF rules.
- Make sure important pages render without JavaScript.
- Pair crawl data with citations and referrals before drawing conclusions.
The bot mix in your logs will keep moving. Meta went from background noise to the top of the chart within a quarter, and ChatGPT-User can lose 92% of its requests on a site overnight. The only stable plan is to measure continuously and decide per bot, not per headline.
