Key takeaways
- Promptwatch Data measured Meta-WebIndexer at about 2.2% of tracked AI crawler requests in mid-July 2026 and 37.8% on August 9. That is roughly 17x in under a month, from one crawler.
- Meta says this crawler improves Meta AI search quality, and that allowing it helps Meta cite and link to your content. It respects robots.txt.
- Not every Meta bot behaves the same. Meta-ExternalFetcher and FacebookExternalHit may bypass robots.txt, so one rule will not cover them all.
- Small sites have felt it. Fast Company reported an artist's site hitting 100% CPU and crashing from Meta bots.
- Decide per crawler, write the robots.txt rule, then check your logs to confirm it holds. Allow up to 24 hours for changes to take effect.
What actually happened
In mid-July 2026, Meta-WebIndexer was a background presence. In Promptwatch's crawler logs it accounted for roughly 2.2% of AI crawler requests. By August 9 it was 37.8%. More than a third of all tracked AI crawler traffic, from a single user agent.
The Promptwatch Data report on Meta-WebIndexer lays out the timeline:
- The share hovered around 2% until July 17.
- It spiked to about 23% between July 20 and 22.
- It surged again from August 5 and peaked at 37.8% on August 9.
- August 10 was a partial day, so treat it with caution.
On August 6, the developer @levelsio posted that Meta was scraping his sites "heavy heavy heavy" and tripped load alerts on one VPS. He said Meta staff told him Meta is building its own web index so its AI does not have to search through Google. Nobody at Meta has confirmed that, so take it as a rumor that happens to fit the data.

How to read the 37.8% number
This is where people will misquote the stat, so be careful. The share is meta-webindexer requests divided by requests from all the AI crawlers Promptwatch tracks, identified by user agent. Meta's other crawlers count in the denominator but not the numerator. Googlebot, Bingbot, FacebookExternalHit and humans are excluded entirely.
So it does not mean 38% of your site traffic is Meta. It means that among AI crawler requests, this one bot grew from a rounding error to the biggest slice. Also, it is user-agent based, so spoofed requests may be mixed in.
A second Promptwatch Data report, AI crawler traffic over three months, uses IP-verified requests from OpenAI, Anthropic, Google, Perplexity and Mistral. It does not list Meta, and the two measures are not directly comparable. In that report OpenAI held 79.8% of verified requests in Aug 31 to Sep 6, down from 94.8% in Jun 8 to 14. Don't put the two numbers side by side on one chart.
I don't know whether Meta-WebIndexer's share held after August 9. The report doesn't say, and I haven't seen follow-up data. Check your own logs rather than assuming the peak is the new normal.
Meta's crawler family, sorted
Meta documents its crawlers on its web crawlers page for developers, last updated May 21, 2026. Here is the version you can actually act on.
| User agent | What Meta says it does | Honors robots.txt? |
|---|---|---|
| meta-webindexer | Improves Meta AI search result quality; helps Meta AI cite and link to you | Yes |
| meta-externalagent | Trains foundation AI models or indexes content directly to improve products | Yes |
| Meta-ExternalAds | Improves advertising and business products | Yes |
| meta-externalfetcher | Fetches individual links at a user's request, supports agentic AI functions | May bypass |
| FacebookExternalHit | Generates link previews on Facebook, Instagram, Messenger | May bypass for security or integrity checks |
The thing to notice: the indexer and the training crawler are different bots with different purposes. Blocking one does not block the other. The Digital Applied analysis of Meta's October 30 terms makes the same point, and I agree with its advice to decide per crawler, not per company.
Why you might actually want it
Meta's own wording is that allowing Meta-WebIndexer "helps us cite and link to your content in Meta AI's responses." That is the upside: a chance to be cited in an assistant that sits inside Facebook, Instagram, WhatsApp and Messenger.
PPC Land adds a wrinkle for marketers. The interactions behind Meta AI also feed Meta's advertising personalization, so one robots.txt line matters on both the organic and paid sides. Whether that bothers you is a business call.
My take: for a brand that wants AI visibility, blocking the search crawler by reflex is a bad default. Blocking the training crawler while allowing the search one is a defensible position. Meta AI is a real surface now. Whether it sends meaningful traffic to your site is something you can measure instead of guess at.
The cost side: when bots hurt
Fast Company reported on California artist Chelsea Paulus, whose site was hit by Meta-ExternalAgent stuck in a loop on the remove-from-cart function. More than a hundred hits per minute, about 550,000 hits over a few days, and CPU at 80% with her host warning about 100%. She blocked that bot. Two days later Meta-WebIndexer showed up and pushed her CPU to 100%, crashing the site.

That is one anecdote, not a statistic. But the pattern is familiar to anyone who has watched logs. A Reddit TechSEO thread analyzing 48 days of server logs called Meta the most aggressive AI crawler by volume, and noted that if you deploy changes during off-peak US hours, AI bots may be the first to see them.
Signs your site is paying for this:
- Slowdowns at odd hours, when customers are not active.
- Server load climbing while human traffic stays flat.
- Bandwidth growing faster than visitor counts.
- Bills that scale by request going up for no clear reason.
Proof Digital's 2026 bot audit cites a figure that 57% of website requests in June 2026 came from automated traffic. Treat that as a ballpark from one source, but the direction matches what site owners report.
The October 30 terms change
A separate issue, easy to confuse with the crawler surge. Nieman Lab reported on October 15, 2026 that Meta's updated commercial terms take effect October 30. Business accounts that keep using Facebook, Instagram or WhatsApp consent to Meta scraping websites associated with those accounts. Three legal experts told Nieman Lab the wording could be used to argue Meta can scrape news sites and use the content to train AI products.

Meta's position, per the same article, is that the terms will not change how it crawls publishers, and that content owners can still manage access using robots.txt. So robots.txt is still your control surface. The terms matter most for publishers with licensing ambitions, who should decide deliberately instead of drifting into consent.
What to do: a practical sequence
1. Look at your logs first
Before touching robots.txt, find out whether this affects you. Search server or CDN logs for meta-webindexer, meta-externalagent, meta-externalfetcher and FacebookBot. Note request counts, which paths get hit, and error rates.
If you only have raw access logs, a quick check works:
grep -i "meta-webindexer" access.log | wc -l
grep -i "meta-externalagent" access.log | wc -l
For ongoing visibility, AI crawler analytics helps. Promptwatch's AI crawler logs are IP-verified, which filters out spoofed bots, and connect through Cloudflare, Vercel, Fastly, other CDNs, a WordPress plugin or raw logs. Tools like Botify, Oncrawl and FandangoSEO also handle log analysis from the technical SEO side.

2. Decide per crawler
A reasonable starting point for most businesses:
| Goal | Meta-WebIndexer | Meta-ExternalAgent | FacebookExternalHit |
|---|---|---|---|
| Want Meta AI citations, wary of training | Allow | Block | Allow |
| Want maximum AI exposure | Allow | Allow | Allow |
| Server under strain, low AI value | Limit by path | Block | Allow |
| Publisher with licensing plans | Decide deliberately | Decide deliberately | Allow |
Keep FacebookExternalHit working. It generates link previews when someone shares your page, and breaking it makes your shared links look bad. Per Meta's docs it needs gzip and deflate support, Open Graph tags within the first 1 MB, a fast response, and handling of the Range header. Allowlisting by IP is more secure than by user agent, and the Sharing Debugger forces a recrawl.
3. Write the robots.txt rules
Meta prefers robots.txt over non-standard signals such as NoAI tags. Examples:
# Allow the AI search crawler, block the training crawler
User-agent: meta-webindexer
Allow: /
User-agent: meta-externalagent
Disallow: /
To limit rather than block, use targeted paths:
User-agent: meta-webindexer
Disallow: /cart/
Disallow: /checkout/
Disallow: /search
That last pattern would have helped in the cart-loop story. Keep the crawlers out of dynamic, stateful URLs. Meta's own docs example only shows rules for meta-externalagent and meta-externalads, so don't assume the same snippet covers the indexer. Write its rule explicitly.
Two cautions. Allow up to 24 hours, because crawlers cache robots.txt. And remember that meta-externalfetcher and FacebookExternalHit may bypass these rules, so robots.txt alone will not protect a fragile server.
The free robots.txt generator for AI and the AI bots and crawlers directory are handy for checking user agent strings.
4. Handle load at the infrastructure level
PPC Land notes that site owners in a March 2026 Cloudflare forum thread described the crawler arriving from many rotating IPv6 addresses at once. Address-level blocking becomes whack-a-mole. Use user-agent or token-based rules at your CDN or WAF instead, and rate limit rather than outright block if you still want citations.
Also:
- Cache heavy pages so crawlers hit the edge, not your origin.
- Make sure cart, session and filter URLs return fast or are disallowed.
- Budget for sustained crawl load if your host bills per request.
- Contact [email protected] if behavior looks broken.
I wouldn't trust a bot to be polite just because its docs say so. Verify in the logs after each change.
5. Measure what you get back
Crawling is cost. Citations and visits are the return. Track whether Meta AI mentions or cites your brand, and whether AI-referred traffic shows up in analytics. If a provider is large in the overall crawler mix but near zero in your logs, you may be blocked by accident. The reverse also applies: lots of crawling and no citations means your content may not be getting used.
Promptwatch covers Meta AI as one of the real interfaces it monitors, alongside ChatGPT, Gemini, Perplexity and others, and pairs prompt tracking with crawler logs and visitor analytics so you can see whether crawling turns into citations and visits. Its guide on attributing AI traffic is a good starting point. If you want to compare other options, the directory at bestgeosoftware.com lists GEO platforms, and Otterly.AI and Peec AI are lighter prompt-monitoring choices without crawler logs.

How this fits the wider crawler picture
Meta is not the only crawler shifting. Promptwatch Data also tracked ChatGPT's use of the site: operator jumping from about 0.4% to about 17% of fanout queries on August 8, 2026, and Claude's citation crawler growing through early 2026 in its crawler visits report. The takeaway is that AI crawler mixes move fast. A robots.txt written in January can be wrong by August.
For agencies running this audit for clients, the same process applies. If you need help with technical SEO and AI search strategy, 1001 SEO Media works on exactly this kind of crawler and GEO audit.
Checklist
- Pull 30 days of logs and count Meta user agents.
- List which Meta crawlers you want to allow, limit or block.
- Write explicit robots.txt rules for meta-webindexer and meta-externalagent.
- Keep FacebookExternalHit working for link previews.
- Disallow cart, checkout, search and filter paths.
- Add CDN rules by user agent, not IP.
- Wait 24 hours, then re-check logs.
- Review monthly, since the share can swing from 2% to 38% in weeks.
The surge itself is a fact. The reason behind it, a Meta search index, is still unconfirmed. Plan for the crawling you can see in your own logs, not the story behind it.


