Key takeaways
- Anthropic documents three robots, not two: ClaudeBot (training), Claude-User (fetches triggered by a person's question) and Claude-SearchBot (search quality). Each has its own robots.txt token.
- A user-agent string is a claim, not proof. Verify the source IP against Anthropic's published list at claude.com/crawling/bots.json before you trust a hit.
- Claude-User is the closest thing to a live signal that a real person asked Claude about your page. Whether Claude Code's own fetches show up under that name is not confirmed by Anthropic, so treat that link as likely but unproven.
- Blocking ClaudeBot does not block Claude-User or Claude-SearchBot.
- Logs measure access. They do not tell you whether Claude cited or recommended you.
Why this is worth looking at
Most people check their server logs for Googlebot and call it a day. Then someone on the team says "Claude can't read our docs" or "Claude is hammering us," and nobody has data.
Your access log is the only first-party record of what machines actually requested. A dashboard that counts user-agent strings is not the same thing. Empact Partners ran this check on twelve of their own domains behind one Cloudflare account for 23 to 29 October 2026. Of 13,442 requests that named an AI crawler, 6,438 passed the address check. And of 2,071 that claimed to be an assistant's live fetch, only 2 passed. A name-counting dashboard would have overstated live assistant fetches roughly a thousand times.

So the working order is: match the claimed bot, understand its purpose, verify it, then analyse what it did.
The three Anthropic robots
Anthropic's own crawler documentation lists three, and some third-party guides only mention two. Here is what each one is for.
| Robot | What Anthropic says it does | Effect of blocking it |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to model training | Your future content is excluded from training datasets |
| Claude-User | Fetches pages when individuals ask Claude questions | Prevents retrieval in response to a user query, may reduce visibility for user-directed web search |
| Claude-SearchBot | Navigates the web to improve search result quality | Prevents indexing for search optimization, may reduce visibility and accuracy in search results |
The split matters because the same site can reasonably want different answers for each. You might be fine with live retrieval and not with training.
There is also a separate page for Anthropic-hosted browsers. Agents such as Claude-Agent, used in Claude Duet, identify themselves differently, so don't expect them under the three names above.
One more name you may see: claude-web. Some guides call it a legacy alias for Claude-User. Anthropic's page doesn't mention it, so I'd treat it as old and unverified. Log it, flag it, don't build policy on it.
Where Claude Code fits
This is the part people get wrong, so let me be careful.
Claude Code is a coding agent that runs on a developer's machine. It can fetch web pages while it works, for example reading your API docs to write an integration. The obvious question is whether those fetches carry the Claude-User name.
I could not find an Anthropic statement that confirms it. What exists is circumstantial. Promptwatch Data's report on Claude citation crawler visits says the crawler may visit cited pages when someone asks Claude a question or uses Claude Code, and it names the bot Claude-User. That is a reasonable reading, but it is the reporting of a third party, not documentation.
My suggestion: assume Claude-User hits on docs, SDK pages and pricing pages include some Claude Code activity, and don't claim it as fact in a report. If you need certainty, test it. Ask Claude Code to read a URL you control that nothing else links to, then look for that path in your log.
What the growth looks like
In Promptwatch's data on Claude citation crawler visits, covering Dec 10, 2025 to Apr 18, 2026 on a 7-day rolling average, Claude-User grew from about 30 visits a day (roughly 0.04% of tracked AI citation crawler traffic) to several thousand a day by mid-April. That is more than 100x. The daily share peaked at 1.73% in mid-April and sat at 1.09% on April 18, with a 0.45% share over the whole 130 days.
Keep that in proportion. Claude is still around 1% of tracked citation crawler traffic. For the wider mix, Promptwatch's AI crawler traffic report (June 8 to Sept 6, 2026, IP-verified requests only) shows OpenAI at 79.8% in the week of Aug 31 to Sept 6, down from 94.8% in early June. So other providers, Anthropic included, are taking a growing share, but from a small base.
Which log fields you need
You don't need a special analytics product to start. You need request-level logs that keep enough per line.
| Field | Why it matters |
|---|---|
| Timestamp | Lets you line hits up with content updates |
| Source IP | The only thing you can actually verify |
| Hostname | Separates subdomains, which robots.txt treats separately |
| Path | Shows what was read |
| HTTP status | Tells you whether the fetch worked |
| User agent | The claimed identity |
| Bytes sent | Spots heavy or truncated responses |
| Response time | Flags slow pages that bots may abandon |
Nginx's access log module and Fastly's logging can both record these. If your host strips the IP or the user agent, fix that first, because everything else depends on it.

Spotting the bots with awk and grep
Match on the stable token, not the full user-agent string. Full strings change; the names mostly don't. I haven't verified exact full UA strings for the Anthropic bots, which is another reason to match on tokens.
A daily count per bot from an Nginx or Apache combined log:
awk '/ClaudeBot|Claude-User|Claude-SearchBot/ {
split($4, d, ":"); date = substr(d[1], 2)
for (b in bots) if ($0 ~ b) hits[date " " b]++
}
BEGIN { split("ClaudeBot Claude-User Claude-SearchBot", bots, " ") }
END { for (k in hits) print k, hits[k] }' access.log | sort
Then see which URLs each one asks for:
grep "Claude-User" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30
grep "ClaudeBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30
And check the status codes, because a bot that gets 403s is not reading anything:
grep "Claude-User" access.log | awk '{print $9}' | sort | uniq -c
Field positions assume the default combined format. Adjust if your format differs.
Verify before you trust
Anthropic publishes its crawler IPs at https://claude.com/crawling/bots.json. If a request's source IP is on that list, it came from Anthropic. If it isn't, the user agent is just a string someone typed.
The workflow is short:
- Pull the IPs out of your Claude-matching lines.
- Compare them with the published list.
- Split the results into verified and unverified, and analyse only the verified set.
- Look at what the unverified ones asked for. Requests for password files or config paths wearing an AI name are scanners, and in Empact's sample none of those passed the check.
Some caution on the flip side. Anthropic says blocking by IP address may not work reliably and can stop it from reading your robots.txt. So use the IP list to verify, not as your blocking mechanism. Use robots.txt for policy.
What the patterns mean
ClaudeBot
Steady, broad coverage across many URLs looks like training-oriented collection. Its URL list is your crawl footprint. It tells you what could end up in training data, not what people are asking about.
Claude-User
Narrow, bursty and tied to specific URLs. A cluster of hits on one page shortly after you publish or update it suggests a person just pointed Claude at it. Pages with repeat Claude-User traffic are the ones people actually want Claude to read: docs, pricing, comparison pages, changelogs.
If some of that is Claude Code, you'll tend to see technical pages dominate. That is a hint, not a measurement.
Claude-SearchBot
This is about search result quality, so expect it to look more like indexing than reading on demand. If it is near zero while the others are active, check robots.txt and any WAF rule first.
robots.txt: three tokens, three decisions
Each bot is controlled separately, and you need to repeat the rules per subdomain. Anthropic's example is User-agent: ClaudeBot followed by Disallow: /. A policy that allows live retrieval but declines training looks like this:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
Crawl-delay is supported as a non-standard extension, for example Crawl-delay: 1, if the bots are heavy on a small server.
As of Oct 30, 2026, Anthropic also honors a machine-readable "no AI training" preference signal based on the IETF aipref draft, rolling out across services over time. If your robots.txt names ClaudeBot, Anthropic follows that rule over the general signal. If it doesn't, it follows the general signal.
The pitfall is the one that bites most often: disallowing ClaudeBot does nothing to Claude-User or Claude-SearchBot. People block one name, assume Claude is shut out, and then wonder why Claude-User keeps showing up. Or the reverse, and they block everything and lose retrieval visibility.
Don't confuse these logs with Claude Code's own logs
Searching for "Claude Code logs" mostly turns up something different: the records Claude Code keeps on the developer's own machine. They are not your server logs, and they tell you nothing about your site's traffic. Dash0's guide lists five record types: session transcripts, debug logs, MCP server logs, prompt history and OpenTelemetry events.

If you are the developer, session transcripts live under ~/.claude/projects/<project>/<session-id>.jsonl and, per Dash0, are kept 30 days by default. That is useful for seeing what Claude Code fetched on your behalf. It is a different question from what your own server saw.
What logs can't tell you
A log line proves access. It does not prove the page was used, cited or recommended. A Claude-User fetch is a good sign, but a fetch can happen and the answer can still skip you.
So it helps to pair the two views: logs for access, and something that tracks what AI answers actually say and cite. Tools like Promptwatch cover both sides, with IP-verified AI crawler logs and citation tracking in one place. Its crawler analytics set up through Cloudflare, Vercel, Fastly or raw logs.

If you'd rather stay with a technical SEO crawler and log analyzer, FandangoSEO is in that category.

For a wider look at options, the GEO software directory lists tools in this space.
A weekly routine that takes 20 minutes
- Pull seven days of logs. Keep timestamp, IP, host, path, status and user agent.
- Count hits per Anthropic token, per day.
- Verify IPs against bots.json. Discard the rest from your analysis, but skim them for scanners.
- List the top 20 URLs for Claude-User. Note which are new, and which are your highest-value pages.
- Check status codes. Any 403, 429 or 5xx on these bots is a fix to ticket.
- Compare your mix with a public benchmark. The Promptwatch report advises that if a provider is large in the report but near zero in your logs, you are probably blocked or unreachable for that provider. Check robots.txt, CDN and WAF rules.
- Baseline monthly. Promptwatch's Claude report recommends baselining Anthropic user agents and reviewing them monthly, and given 100x growth in four months, that cadence is sensible.
Troubleshooting
- Zero Claude hits: look at robots.txt per subdomain, then WAF bot rules, then whether your CDN logs are complete.
- Lots of ClaudeBot, no Claude-User: normal for many sites. Live fetches depend on people asking about you.
- Claude hits from unlisted IPs: spoofed. Block at the WAF if they cause load; don't count them.
- Genuine crawler problems: Anthropic asks you to email [email protected] from an address that includes the domain you are asking about.
The short version
Match on three tokens, verify against Anthropic's IP list, and read each bot for what it is. ClaudeBot is training, Claude-SearchBot is search quality, Claude-User is live retrieval and probably includes some Claude Code activity. Block and allow them separately. Then check whether access turns into citations, because the log can't do that for you.