Meta-WebIndexer Went From 2% to 38% of AI Crawler Traffic: Should You Allow It in 2026?

Key takeaways

• Meta-WebIndexer rose from about 2.2% of tracked AI crawler requests in mid-July 2026 to 37.8% on August 9, a roughly 17x jump, according to Promptwatch Data.
• Meta says it crawls to improve Meta AI search results, and that allowing it in robots.txt helps Meta cite and link to your content.
• Meta runs several crawlers with different jobs. A blanket block of everything breaks Facebook link previews, so decide per crawler.
• For most sites the practical default is selective: allow Meta-WebIndexer and FacebookExternalHit, and block Meta-ExternalAgent if training use bothers you.
• Check your logs and your bandwidth bill first. Some site owners report millions of Meta requests per month.

What happened

In mid-July 2026, Meta-WebIndexer made up roughly 2.2% of the AI crawler requests in Promptwatch's logs. The share hovered around 2% until July 17, spiked to about 23% between July 20 and 22, then climbed again from August 5 to a peak of 37.8% on August 9. That is about 1 in 50 AI crawler requests turning into more than 1 in 3. It made Meta-WebIndexer the heaviest AI crawler in those logs. The full daily chart is in Promptwatch's Meta-WebIndexer report.

A few caveats before you panic or celebrate:

• The number is a share of the AI crawler requests Promptwatch tracks, identified by user agent. It is not a share of your total site traffic.
• Meta's other crawlers (meta-externalagent, meta-externalfetcher, FacebookBot) count only in the denominator. Googlebot, Bingbot, FacebookExternalHit and human visitors are excluded.
• Promptwatch's separate AI crawler traffic report covers IP-verified requests from OpenAI, Anthropic, Google, Perplexity and Mistral, and does not list Meta. The two reports use different methods, so don't compare their percentages directly.

Is Meta building a search index?

The pattern looks like what building a web index from scratch looks like. The explanation circulating is that Meta wants its own index so Meta AI doesn't depend on Google or Bing. The source for that is a post on X by @levelsio on August 6, 2026, who said Meta staff told him Meta is "ALLEGEDLY" building its own search engine. He also said Meta's scraping triggered load alerts on one of his servers. That is secondhand and unconfirmed, and I could not find an official Meta statement confirming a search index.

What Meta does say, in its own crawler documentation, is that Meta-WebIndexer "navigates the web to improve Meta AI search result quality." That part is official.

Meta's crawlers, and what each one does

The name soup is the first problem. Meta documents several crawlers, and treating them as one thing leads to bad decisions. The details below come from Meta's web crawler documentation.

Two things stand out. The surge in Promptwatch's data comes from the indexing crawler specifically, not the training or link-fetching bots. And two of the crawlers can ignore robots.txt, so a Disallow rule will not stop them. If you need hard control there, you need CDN or firewall rules.

The case for allowing Meta-WebIndexer

Meta's stated reason is simple: allowing the crawler helps Meta cite and link to your content in Meta AI's answers. If you care about showing up in AI answers at all, this is the same logic as letting OAI-SearchBot or PerplexityBot in.

Blocking has a cost that is easy to miss. If Meta is indeed building an index, whether your pages were crawled during the build-out could decide whether you can be cited when it launches. That is speculation built on an unconfirmed report, so weigh it accordingly. But the downside of allowing the indexer is mostly server load, while the downside of blocking is possible invisibility on a surface with Meta's reach.

Who should lean toward allowing it:

• Brands that want to be recommended in AI tools more than they want to control training use
• Publishers and ecommerce sites whose pages exist to win buyers
• Sites with solid hosting where extra crawl load is a rounding error

The case for blocking or limiting it

The complaints are loud and mostly about cost.

On Reddit's r/webdev, one owner reported 7.9 million Meta AI crawler requests in 30 days and 900+ GB of bandwidth before noticing. Others in the thread reported 741,900 and 11 million requests in a month. Those reports are anecdotal, and they mostly involve meta-externalagent rather than the indexer. A Cloudflare community thread describes meta-externalagent and meta-externalfetcher generating excessive repeated requests. A Towards AI writer reported paying about $25 a month extra for crawler traffic on a community site.

So the honest summary is this. Meta-WebIndexer's share is new and large, but the loudest cost stories are about other Meta bots. Either way, I couldn't find any rate-limit or Crawl-delay guidance from Meta, which means your options are block, allow, or throttle at your own edge.

Who should lean toward blocking or limiting:

• Sites billed per request or per GB, like small VPS setups and metered serverless hosting
• Sites with expensive dynamic pages, such as faceted search or large catalogs
• Publishers with contractual or licensing reasons to restrict AI training use

There is also a legal and contractual angle for some teams. One Reddit commenter mentioned a data processing agreement concern. A third-party guide from Single Grain also says Meta's new commercial terms include a broad crawling clause, with the effective date shifting from October 30 to November 20, 2026. I could not verify that against Meta's terms page, so have your legal team read the source directly if it matters to you.

A selective policy that works for most sites

Here is the default I would start with, which matches the selective approach in Single Grain's guide:

1. Allow Meta-WebIndexer, so Meta AI can cite you.
2. Allow FacebookExternalHit, so link previews keep working.
3. Decide on Meta-ExternalAgent based on your view of training use.
4. Leave Meta-ExternalFetcher alone unless it is hurting you, and know that robots.txt won't reliably stop it.

Robots.txt examples

Block only the training and indexing crawler:

User-agent: meta-externalagent
Disallow: /

Block the indexer too, if server cost matters more than Meta AI citations:

User-agent: meta-externalagent
Disallow: /

User-agent: meta-webindexer
Disallow: /

Allow the indexer explicitly while keeping the rest of your rules:

User-agent: meta-webindexer
Allow: /

Meta says changes can take up to 24 hours to apply because crawlers may cache robots.txt. It also says it does not use non-standard formats such as NoAI tags, so don't bother with those. For questions, Meta lists [email protected].

Keep link previews working

If you do block aggressively at the firewall, check that FacebookExternalHit still gets through. Meta's requirements for it are practical:

• Support gzip and deflate encodings
• Put Open Graph tags within the first 1 MB of the response
• Respond within a few seconds
• Allowlist by user agent, or by IP address, which Meta says is more secure

The Sharing Debugger tells you quickly whether a URL renders properly.

Check your logs before deciding

Don't guess. Pull a month of server or CDN logs and count requests by user agent for meta-webindexer, meta-externalagent, meta-externalfetcher and FacebookBot. Note bytes transferred, not just request counts, since heavy pages matter more than hit totals.

Two cautions:

• User agents can be spoofed. Verify against Meta's published IP ranges before you block on user agent alone. I did not retrieve the ranges, so pull them from Meta's documentation.
• If a provider is a big share of the overall mix but near zero in your logs, you are probably blocked or unreachable for that provider. That applies to Meta as much as to OpenAI.

For log analysis you can use your CDN's dashboards, a crawler like Screaming Frog SEO Spider alongside a log file analyzer, or a dedicated log analysis tool.

Platforms built for AI visibility also surface this. Promptwatch logs IP-verified AI crawler requests, filters out spoofed bots, and shows which pages each crawler read and whether they were then cited.

And if you want a wider view of options, the GEO software directory at bestgeosoftware.com lists more tools in this category.

Decision table

Throttling instead of blocking

Robots.txt is all or nothing, and Meta has not published Crawl-delay guidance. If load is the only concern, rate limit at the edge. Cloudflare, Fastly and most CDNs let you set per-user-agent rules, and you can cache aggressively for crawler traffic so repeated hits don't reach your origin. The Cloudflare thread above is a good place to see what other site owners tried.

A reasonable order of operations:

1. Confirm the traffic is genuinely Meta, by IP.
2. Cache the pages being hit repeatedly.
3. Add a rate limit for the offending user agent.
4. Only then reach for a robots.txt block.

What to do next

This week: pull your logs, count Meta requests by crawler, and check your robots.txt for accidental blanket blocks. Then set a policy per crawler instead of one rule for all of Meta. Revisit in a month, since the numbers in this story moved from 2% to 38% in under 30 days and could move again.

If you want hands-on help setting an AI crawler policy alongside the rest of your AI search work, that is the kind of work 1001 SEO Media does as part of its technical SEO and AI search services.

For broader context on what AI engines actually cite, see Promptwatch's citation research hub. One finding worth knowing: in Promptwatch's data for Sep 28 to Oct 4, 2026, ChatGPT got only 0.46% of its citations from social platforms, while Google AI Overviews got 12.2%. Each AI surface behaves differently, so each crawler deserves its own decision.