Key takeaways
- Bots now make up more of Cloudflare's network traffic than humans do. Radar has shown readings bouncing between roughly 57% and 64% bot traffic through 2026, and AI agents are a big part of why.
- Cloudflare AI Crawl Control, Ahrefs Bot Analytics, and Semrush's "Blocked from AI Search" report approach the same problem from three different angles: crawler management, log-level analytics, and robots.txt auditing.
- An estimated 27% of websites block at least one major AI crawler by accident, usually because robots.txt was written for Googlebot years ago and nobody updated it for GPTBot, ClaudeBot, or PerplexityBot.
- Meta-WebIndexer went from 2% to nearly 38% of all tracked AI crawler requests between mid-July and early August 2026, which means a crawler most site owners have never heard of is suddenly the heaviest one hitting their servers.
- None of these three tools alone tells you whether your content actually gets cited in AI answers. Crawling is a precondition, not a result, so pair bot-detection tools with something that tracks citations and AI-referred traffic.
Why bot detection suddenly matters
I'll be honest, "bot traffic" used to be a security topic, not an SEO one. You worried about credential-stuffing attacks and scraper farms, not whether your content was reachable by a crawler you wanted to visit. That changed fast in 2026.
Cloudflare CEO Matthew Prince posted on X in June 2026 that bot traffic had passed human traffic on Cloudflare's network for the first time ever, about 18 months earlier than he'd predicted. Cloudflare Radar's live dashboard has shown the bot share drifting between roughly 57% and 64% of HTTP requests to HTML pages since then, depending on the day you check. That's not a rounding error. Most of the growth is agentic: AI assistants fetching pages live to answer a question, and AI search crawlers pre-indexing content for ChatGPT search mode, Perplexity, and Google's AI Overviews and AI Mode.
The practical problem this creates is simple and a little embarrassing: a robots.txt file written five or ten years ago, when "bots" meant Googlebot and maybe Bingbot, can silently block crawlers it was never designed to think about. ClaudeBot, GPTBot, ChatGPT-User, and PerplexityBot all use distinct user-agent strings, and a blanket disallow rule or an aggressive security plugin can wall them out without anyone noticing, right up until someone asks why the brand never shows up in ChatGPT answers.
The three approaches: manage, log, audit
Cloudflare, Ahrefs, and Semrush aren't really competing products here. They solve adjacent but different problems, and most serious SEO teams end up using some combination of all three.
Cloudflare AI Crawl Control: manage the crawlers at the edge
Cloudflare's tool, formerly called AI Audit and now renamed AI Crawl Control, sits at the network edge and gives you direct control over which AI crawlers can reach your site, available on every plan including the free tier.
It does four things. It lets you set granular allow or block rules per crawler. It gives you a dashboard of crawler activity and request patterns. It tracks robots.txt compliance and flags any crawler that's ignoring your directives, which is useful because not every bot respects robots.txt even when it says it does. And it includes Pay Per Crawl, a monetization feature where you set a flat price per request and AI companies either pay, get blocked, or get allowed through for free, depending on how you configure it.
Beyond the crawlers that identify themselves honestly, Cloudflare's separate Bot Management product (not the free AI Crawl Control tool) uses machine learning, behavioral analysis, and fingerprinting to catch bots that disguise their user agent to look human. That's the enterprise-only layer, and pricing isn't public, but industry estimates put Cloudflare Bot Management at around $5,000 a month minimum with an annual commitment, and full enterprise WAF-plus-bot-management deployments commonly running $8,000 to $25,000 a month.
Cloudflare Radar, the public-facing companion to all this, also breaks AI bot traffic down by purpose: Training, Search, User action, and Undeclared. The most recent breakdown I've seen puts Training traffic at around 80% of AI crawling, with User action and Undeclared combined under 5%. ChatGPT-User alone accounted for nearly three-quarters of that User action traffic, and it follows a clear daily usage cycle, which makes sense since it's triggered by actual people asking ChatGPT questions in real time, not a bot crawling on a schedule.
Ahrefs Bot Analytics: the log-level detail Cloudflare doesn't show
Ahrefs' Bot Analytics, free during its 2026 beta and included on all paid plans, works off server-side CDN logs rather than JavaScript tracking, so it captures bot activity that a JS-based analytics tool like GA4 would never see (bots don't execute JavaScript).
It sorts traffic into more than a dozen categories, not just "AI crawler" as a blob but AI Assistant, AI Search, Search Engine, Security, Advertising, Monitoring, Service Agent, Social Media, SEO Tool, and a few more. Six reports break this down: an overview, individual bot identification (so you see Googlebot, GPTBot, and ClaudeBot as separate lines rather than lumped together), category rollups, which pages got hit, status codes returned, and site structure. There's also a one-click "AI bots" filter and a Page Inspect feature that shows what content AhrefsBot specifically crawled on a given URL.
Setup requires a log source. Cloudflare users need Logpush, which unfortunately requires Cloudflare's Enterprise plan, though a Cloudflare Worker gets around that limitation on any plan including free. Vercel, Amazon CloudFront, and Fastly are also supported.
The stat that stuck with me from Ahrefs' own research: AI bots accounted for nearly 25% of all bot requests by mid-2025, and that share has clearly kept climbing given everything else we know about 2026 crawler volume. Cloudflare separately estimates 53% of all crawler traffic is "wasted effort," meaning it's hitting pages that will never generate a citation or a click.
Semrush Site Audit: catching the robots.txt mistake before it costs you
Semrush takes a third angle. Inside Site Audit's Crawled Pages report, there's a specific "Blocked from AI Search" section that shows how many of your audited pages are currently inaccessible to AI crawlers because of robots.txt rules.
What I like about this framing is that it splits AI-related bots into four buckets rather than treating them as one category: assistant bots like ChatGPT-User and Claude-User that fetch content live in response to a specific user question, search crawlers like OAI-SearchBot and PerplexityBot that pre-index content for AI search modes, pure training bots, and traditional search engine bots. That distinction matters because blocking a training bot is a defensible choice many publishers make deliberately, while blocking ChatGPT-User means you've opted out of conversational answers entirely, probably without meaning to.
Semrush frames this as a prerequisite check: fix the robots.txt blocking issue first, because no other AI visibility optimization works if the crawler can't reach the page in the first place. That's a fair point, and it's the same logic behind Semrush's broader argument that bot traffic now exceeds human traffic, so unintentionally blocking AI crawlers has become a bigger risk than it used to be.
Comparison: Cloudflare vs. Ahrefs vs. Semrush
| Tool | What it actually does | Data source | Free tier | Starting paid price |
|---|---|---|---|---|
| Cloudflare AI Crawl Control | Allow/block AI crawlers at the edge, monitor compliance, monetize with Pay Per Crawl | Edge network logs | Yes, included on free plan | Bot Management (the fingerprinting layer) is Enterprise-only, ~$5,000+/mo |
| Ahrefs Bot Analytics | Server-side log analysis across 12+ bot categories, per-page crawl detail | CDN/server logs (Cloudflare Worker, Vercel, CloudFront, Fastly) | Yes, free during 2026 beta | Included in paid Ahrefs plans starting at $29/mo (Starter) |
| Semrush Site Audit | Flags pages blocked from AI crawlers via robots.txt, splits bots by purpose | Semrush's own crawler, checked against robots.txt | No standalone free tier for this feature | Pro at $139.95/mo (or $117.33/mo billed annually) |
None of these three replace each other. Cloudflare is the only one that can actually block or charge a crawler in real time. Ahrefs gives you the most granular forensic detail about what already happened in your logs. Semrush is the fastest way to spot a robots.txt mistake during a routine audit, before you've even thought to look at crawler logs.
The crawler mix keeps shifting, fast
The thing that makes this whole category hard to keep up with is that the relative importance of each crawler changes month to month, not year to year. According to Promptwatch's tracking of IP-verified AI crawler requests, OpenAI's crawlers accounted for 94.8% of verified AI crawler activity in the week of June 8-14, 2026, but had dropped to 79.8% by the week of August 31-September 6, 2026, as other providers gained ground.
Anthropic's Claude crawler is a good example of how fast a "minor" bot can become a real consideration. Promptwatch's data on Claude citation crawler visits shows it went from about 30 visits a day, roughly 0.04% share, in mid-December 2025 to a peak of 1.73% share by April 12, 2026, a hundred-fold increase in four months, with the real inflection point happening in late March when daily visits jumped from the hundreds into the thousands within two weeks.
Then there's Meta. Promptwatch's research on Meta-WebIndexer found it went from about 2.2% of all tracked AI crawler requests in mid-July 2026 to 37.8% by August 9, roughly a 17x jump in under a month, making it the single heaviest AI crawler tracked at that point. Indie developer @levelsio reported in early August that Meta staff had privately confirmed the company is building its own web index so Meta AI doesn't have to depend on Google, and that the scraping was heavy enough to trigger server load alerts on his sites. If your infrastructure bills by request volume, that's the kind of shift that shows up on an invoice before it shows up in a dashboard.
If you want a sense of the full taxonomy these tools are trying to track, it's genuinely long: GPTBot, OAI-SearchBot, ChatGPT-User, and OAI-AdsBot for OpenAI; ClaudeBot, Claude-User, Claude-SearchBot, and claude-web for Anthropic; Google-Extended and Google-Agent for Google; PerplexityBot and Perplexity-User; GrokBot, xAI-Grok, and Grok-DeepSearch for xAI; MistralAI-User and MistralAI-Index; and on the Meta side, meta-webindexer, meta-externalagent, meta-externalfetcher, and FacebookBot. Plus Cohere and DeepSeek have their own crawlers now too. A robots.txt file that only mentions Googlebot is missing basically all of this.
A baseline robots.txt that doesn't shoot you in the foot
The common failure mode isn't malicious, it's just neglect. A good starting point, if you want AI search engines to be able to cite you while still blocking pure-training scrapers you don't want, looks roughly like this:
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
That's a deliberate "selective access" strategy: let in the bots that power AI search and citations, and make a separate, conscious decision about pure-training scrapers like CCBot or Bytespider if you'd rather not feed a general-purpose training set. The diagnostic tip worth remembering is that a wave of HTTP 403 responses for known AI user agents in your server logs means the block is happening at the robots.txt, server, or firewall level, not that the crawler simply isn't interested in your site.
Crawling is not the same as being cited
Here's where I'll push back a little on treating bot-detection tools as the whole solution. Cloudflare, Ahrefs, and Semrush all answer some version of "is the bot reaching my site." None of them tell you whether that crawl turned into an actual citation in a ChatGPT answer, whether your page showed up in a Google AI Overview, or whether any of that traffic converted into a visitor who clicked through to your site.
That's a genuinely different question, and it needs a different kind of tool. This is where platforms built specifically for AI visibility come in, tools like Promptwatch connect the crawler-log side of the equation (which pages AI bots actually read, and whether they hit errors) to the citation side (which of those pages actually got cited, in which AI platform, and whether that citation drove real traffic). Promptwatch's crawler log feature, Agent Analytics, tracks more than 400 AI crawlers in real time and maps the crawl-to-citation path per page, which is the piece that Cloudflare's edge-level blocking and Ahrefs' log analytics don't attempt to do.
For teams focused purely on the bot-management side, the choice is really about where in the stack you want to intervene. If you need to actively block or monetize crawler access, that's Cloudflare's job. If you want forensic detail on exactly what got crawled and how often, Ahrefs Bot Analytics or a dedicated log analyzer like Screaming Frog SEO Spider fits better.
And if your real goal is catching robots.txt mistakes as part of a broader technical SEO audit, Semrush's Site Audit does that well without requiring you to set up log exports first.
If you're trying to go further and tie crawler behavior to actual AI-driven revenue, it's worth browsing the wider category. The GEO software directory at bestgeosoftware.com covers platforms that go beyond monitoring into content optimization, and ai-rank-tools.com is a decent starting point if you're specifically comparing AI rank trackers against each other.
Putting it together
If I were setting this up from scratch in late 2026, I'd start with the cheapest diagnostic step: run a Semrush Site Audit (or check your robots.txt by hand against the crawler list above) to catch any accidental blocks. Then turn on Cloudflare AI Crawl Control, since it's free and gives you a live view of which bots are actually hitting your edge, plus the option to block the ones you don't want. If you need per-page forensic detail, especially for a site getting heavy AI crawler traffic, layer in Ahrefs Bot Analytics using a Cloudflare Worker to avoid the Enterprise Logpush requirement.
Then, separately, decide whether you actually care about citations and AI-referred revenue, not just crawl access. If you do, that's a different budget line and a different tool, and it's the gap none of these three bot-detection products are built to close.