Claude-User and ClaudeBot in Your Server Logs: How to Tell What Claude Code Reads on Your Site (2026)

Anthropic runs three web robots, and they mean very different things when they show up in your logs. Here is how to spot ClaudeBot, Claude-User and Claude-SearchBot, verify them against Anthropic's IP list, and what Claude Code traffic can and cannot tell you.

Key takeaways

  • Anthropic documents three robots, not two: ClaudeBot (training), Claude-User (fetches triggered by a person's question) and Claude-SearchBot (search quality). Each has its own robots.txt token.
  • A user-agent string is a claim, not proof. Verify the source IP against Anthropic's published list at claude.com/crawling/bots.json before you trust a hit.
  • Claude-User is the closest thing to a live signal that a real person asked Claude about your page. Whether Claude Code's own fetches show up under that name is not confirmed by Anthropic, so treat that link as likely but unproven.
  • Blocking ClaudeBot does not block Claude-User or Claude-SearchBot.
  • Logs measure access. They do not tell you whether Claude cited or recommended you.

Why this is worth looking at

Most people check their server logs for Googlebot and call it a day. Then someone on the team says "Claude can't read our docs" or "Claude is hammering us," and nobody has data.

Your access log is the only first-party record of what machines actually requested. A dashboard that counts user-agent strings is not the same thing. Empact Partners ran this check on twelve of their own domains behind one Cloudflare account for 23 to 29 October 2026. Of 13,442 requests that named an AI crawler, 6,438 passed the address check. And of 2,071 that claimed to be an assistant's live fetch, only 2 passed. A name-counting dashboard would have overstated live assistant fetches roughly a thousand times.

Empact Partners article on checking which AI bots really read your site, with a week of Cloudflare request counts

So the working order is: match the claimed bot, understand its purpose, verify it, then analyse what it did.

The three Anthropic robots

Anthropic's own crawler documentation lists three, and some third-party guides only mention two. Here is what each one is for.

RobotWhat Anthropic says it doesEffect of blocking it
ClaudeBotCollects web content that could contribute to model trainingYour future content is excluded from training datasets
Claude-UserFetches pages when individuals ask Claude questionsPrevents retrieval in response to a user query, may reduce visibility for user-directed web search
Claude-SearchBotNavigates the web to improve search result qualityPrevents indexing for search optimization, may reduce visibility and accuracy in search results

The split matters because the same site can reasonably want different answers for each. You might be fine with live retrieval and not with training.

There is also a separate page for Anthropic-hosted browsers. Agents such as Claude-Agent, used in Claude Duet, identify themselves differently, so don't expect them under the three names above.

One more name you may see: claude-web. Some guides call it a legacy alias for Claude-User. Anthropic's page doesn't mention it, so I'd treat it as old and unverified. Log it, flag it, don't build policy on it.

Where Claude Code fits

This is the part people get wrong, so let me be careful.

Claude Code is a coding agent that runs on a developer's machine. It can fetch web pages while it works, for example reading your API docs to write an integration. The obvious question is whether those fetches carry the Claude-User name.

I could not find an Anthropic statement that confirms it. What exists is circumstantial. Promptwatch Data's report on Claude citation crawler visits says the crawler may visit cited pages when someone asks Claude a question or uses Claude Code, and it names the bot Claude-User. That is a reasonable reading, but it is the reporting of a third party, not documentation.

My suggestion: assume Claude-User hits on docs, SDK pages and pricing pages include some Claude Code activity, and don't claim it as fact in a report. If you need certainty, test it. Ask Claude Code to read a URL you control that nothing else links to, then look for that path in your log.

What the growth looks like

In Promptwatch's data on Claude citation crawler visits, covering Dec 10, 2025 to Apr 18, 2026 on a 7-day rolling average, Claude-User grew from about 30 visits a day (roughly 0.04% of tracked AI citation crawler traffic) to several thousand a day by mid-April. That is more than 100x. The daily share peaked at 1.73% in mid-April and sat at 1.09% on April 18, with a 0.45% share over the whole 130 days.

Keep that in proportion. Claude is still around 1% of tracked citation crawler traffic. For the wider mix, Promptwatch's AI crawler traffic report (June 8 to Sept 6, 2026, IP-verified requests only) shows OpenAI at 79.8% in the week of Aug 31 to Sept 6, down from 94.8% in early June. So other providers, Anthropic included, are taking a growing share, but from a small base.

Which log fields you need

You don't need a special analytics product to start. You need request-level logs that keep enough per line.

FieldWhy it matters
TimestampLets you line hits up with content updates
Source IPThe only thing you can actually verify
HostnameSeparates subdomains, which robots.txt treats separately
PathShows what was read
HTTP statusTells you whether the fetch worked
User agentThe claimed identity
Bytes sentSpots heavy or truncated responses
Response timeFlags slow pages that bots may abandon

Nginx's access log module and Fastly's logging can both record these. If your host strips the IP or the user agent, fix that first, because everything else depends on it.

Publive guide on identifying AI crawlers in server logs, showing documented user-agent tokens including ClaudeBot and Claude-SearchBot

Spotting the bots with awk and grep

Match on the stable token, not the full user-agent string. Full strings change; the names mostly don't. I haven't verified exact full UA strings for the Anthropic bots, which is another reason to match on tokens.

A daily count per bot from an Nginx or Apache combined log:

awk '/ClaudeBot|Claude-User|Claude-SearchBot/ {
    split($4, d, ":"); date = substr(d[1], 2)
    for (b in bots) if ($0 ~ b) hits[date " " b]++
}
BEGIN { split("ClaudeBot Claude-User Claude-SearchBot", bots, " ") }
END { for (k in hits) print k, hits[k] }' access.log | sort

Then see which URLs each one asks for:

grep "Claude-User" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30
grep "ClaudeBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30

And check the status codes, because a bot that gets 403s is not reading anything:

grep "Claude-User" access.log | awk '{print $9}' | sort | uniq -c

Field positions assume the default combined format. Adjust if your format differs.

Verify before you trust

Anthropic publishes its crawler IPs at https://claude.com/crawling/bots.json. If a request's source IP is on that list, it came from Anthropic. If it isn't, the user agent is just a string someone typed.

The workflow is short:

  1. Pull the IPs out of your Claude-matching lines.
  2. Compare them with the published list.
  3. Split the results into verified and unverified, and analyse only the verified set.
  4. Look at what the unverified ones asked for. Requests for password files or config paths wearing an AI name are scanners, and in Empact's sample none of those passed the check.

Some caution on the flip side. Anthropic says blocking by IP address may not work reliably and can stop it from reading your robots.txt. So use the IP list to verify, not as your blocking mechanism. Use robots.txt for policy.

What the patterns mean

ClaudeBot

Steady, broad coverage across many URLs looks like training-oriented collection. Its URL list is your crawl footprint. It tells you what could end up in training data, not what people are asking about.

Claude-User

Narrow, bursty and tied to specific URLs. A cluster of hits on one page shortly after you publish or update it suggests a person just pointed Claude at it. Pages with repeat Claude-User traffic are the ones people actually want Claude to read: docs, pricing, comparison pages, changelogs.

If some of that is Claude Code, you'll tend to see technical pages dominate. That is a hint, not a measurement.

Claude-SearchBot

This is about search result quality, so expect it to look more like indexing than reading on demand. If it is near zero while the others are active, check robots.txt and any WAF rule first.

robots.txt: three tokens, three decisions

Each bot is controlled separately, and you need to repeat the rules per subdomain. Anthropic's example is User-agent: ClaudeBot followed by Disallow: /. A policy that allows live retrieval but declines training looks like this:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Crawl-delay is supported as a non-standard extension, for example Crawl-delay: 1, if the bots are heavy on a small server.

As of Oct 30, 2026, Anthropic also honors a machine-readable "no AI training" preference signal based on the IETF aipref draft, rolling out across services over time. If your robots.txt names ClaudeBot, Anthropic follows that rule over the general signal. If it doesn't, it follows the general signal.

The pitfall is the one that bites most often: disallowing ClaudeBot does nothing to Claude-User or Claude-SearchBot. People block one name, assume Claude is shut out, and then wonder why Claude-User keeps showing up. Or the reverse, and they block everything and lose retrieval visibility.

Don't confuse these logs with Claude Code's own logs

Searching for "Claude Code logs" mostly turns up something different: the records Claude Code keeps on the developer's own machine. They are not your server logs, and they tell you nothing about your site's traffic. Dash0's guide lists five record types: session transcripts, debug logs, MCP server logs, prompt history and OpenTelemetry events.

Dash0 guide to the five types of Claude Code logs and where each is stored

If you are the developer, session transcripts live under ~/.claude/projects/<project>/<session-id>.jsonl and, per Dash0, are kept 30 days by default. That is useful for seeing what Claude Code fetched on your behalf. It is a different question from what your own server saw.

What logs can't tell you

A log line proves access. It does not prove the page was used, cited or recommended. A Claude-User fetch is a good sign, but a fetch can happen and the answer can still skip you.

So it helps to pair the two views: logs for access, and something that tracks what AI answers actually say and cite. Tools like Promptwatch cover both sides, with IP-verified AI crawler logs and citation tracking in one place. Its crawler analytics set up through Cloudflare, Vercel, Fastly or raw logs.

Favicon of Promptwatch

Promptwatch

AI search visibility and optimization platform
View more
Screenshot of Promptwatch website

If you'd rather stay with a technical SEO crawler and log analyzer, FandangoSEO is in that category.

Favicon of FandangoSEO

FandangoSEO

Website crawler and log analyzer for technical SEO
View more
Screenshot of FandangoSEO website

For a wider look at options, the GEO software directory lists tools in this space.

A weekly routine that takes 20 minutes

  1. Pull seven days of logs. Keep timestamp, IP, host, path, status and user agent.
  2. Count hits per Anthropic token, per day.
  3. Verify IPs against bots.json. Discard the rest from your analysis, but skim them for scanners.
  4. List the top 20 URLs for Claude-User. Note which are new, and which are your highest-value pages.
  5. Check status codes. Any 403, 429 or 5xx on these bots is a fix to ticket.
  6. Compare your mix with a public benchmark. The Promptwatch report advises that if a provider is large in the report but near zero in your logs, you are probably blocked or unreachable for that provider. Check robots.txt, CDN and WAF rules.
  7. Baseline monthly. Promptwatch's Claude report recommends baselining Anthropic user agents and reviewing them monthly, and given 100x growth in four months, that cadence is sensible.

Troubleshooting

  • Zero Claude hits: look at robots.txt per subdomain, then WAF bot rules, then whether your CDN logs are complete.
  • Lots of ClaudeBot, no Claude-User: normal for many sites. Live fetches depend on people asking about you.
  • Claude hits from unlisted IPs: spoofed. Block at the WAF if they cause load; don't count them.
  • Genuine crawler problems: Anthropic asks you to email [email protected] from an address that includes the domain you are asking about.

The short version

Match on three tokens, verify against Anthropic's IP list, and read each bot for what it is. ClaudeBot is training, Claude-SearchBot is search quality, Claude-User is live retrieval and probably includes some Claude Code activity. Block and allow them separately. Then check whether access turns into citations, because the log can't do that for you.

Share:

© 2026 Toolsolved · Find the best marketig tools · RSS

Toolsolved is an affiliate review site. When you click links to vendors or buy through links on our site, we may earn an affiliate commission at no extra cost to you.

Toolsolved is a review website based on user reviews on Reddit and G2, and on publicly available information. We keep everything as up to date as possible, but pricing and features can change. Always confirm the details with the vendor before purchasing.