Key takeaways
- Prompt tracking tells you what AI says about you. Crawler logs tell you whether AI can even read your site. Most tools only do the first.
- A user-agent string is a claim, not an identity. IP verification checks the request against the operator's published IP ranges and drops spoofed bots.
- Promptwatch is the platform in this guide with documented IP-verified crawler analytics. For others (Peec, Profound, Semrush, Ahrefs, seoClarity, JetOctopus) I could not confirm IP-range verification, so treat that as a question to ask in a demo.
- Log-based tools from the technical SEO side (JetOctopus, seoClarity, Botify, Oncrawl) are worth a look if you already live in logs.
- Whatever you pick, compare your logs against a benchmark. A provider that is big in the benchmark and near zero in your logs usually means you are blocked.
Why crawler logs matter more than another mention counter
A prompt tracker answers one question: was my brand mentioned? Useful, but it skips the layer underneath. Before ChatGPT, Perplexity or Claude can cite a page, some bot has to fetch it. If that bot hits a 403, a WAF challenge or a robots.txt block, no amount of content tuning fixes the problem.
This is also invisible to your normal analytics. Google Analytics runs on JavaScript, and crawlers do not execute it. As the Digital Applied playbook on server logs and AI agents puts it, an autonomous crawler issues an HTTP request, takes your HTML and leaves. No script, no session, no event. The request exists only in your access log.
That same piece splits AI bot traffic by purpose. In its rolling window, 81.7% of AI-bot requests were training-oriented and only 3.2% were live user-action fetches with a person waiting. Those are very different things to block or allow, and you cannot tell them apart without logs.
What IP-verified actually means
Anyone can send a request with the user agent "GPTBot". Plenty of scrapers do exactly that. A log tool that matches only on the user-agent string will happily count them as OpenAI.
IP verification closes that hole. The tool checks the source IP of each request against the ranges the crawler operator publishes. If the IP is not in OpenAI's range, it is not counted as OpenAI, whatever the header claims.
Promptwatch Data describes its own method this way in its AI crawler traffic report: requests come from raw server and CDN logs, and only those whose source IP matches the operator's published ranges are included. Spoofed user agents are excluded. Weeks run Monday to Sunday in UTC, and the shares are 100% bars, so a growing segment means a bigger slice of the mix, not necessarily more requests.
So when you evaluate a vendor, there are really four tiers:
- No crawler data at all (prompt tracking only).
- Crawler data from a JavaScript tag or analytics, which misses most bots.
- Log-based data matched on user agent only.
- Log-based data verified against published IP ranges.
Most of the market sits in tiers 1 and 2. A handful reach 3. Confirmed tier 4 is rarer.
Who has what in 2026
A caveat first. My research confirmed Promptwatch's methodology directly from its published report. For the other vendors I only confirmed that a crawler or log feature exists in places, not how it verifies bots, and I did not verify pricing for any of them. The table reflects that honestly.
| Platform | Crawler or log feature | IP verification | Notes |
|---|---|---|---|
| Promptwatch | Yes, AI crawler logs and Agent Analytics | Yes, documented | Cloudflare, Vercel, Fastly, raw logs and more |
| JetOctopus | Yes, Log Analyzer, 40+ AI bots | Guide covers verifying against spoofing | Technical SEO platform with an AI Search Visibility product |
| seoClarity | Log File Analysis in platform nav | Unconfirmed | Not confirmed as AI-specific |
| Peec AI | Built-in crawler log analysis reported | Unconfirmed | Primarily a prompt tracker |
| Profound | Unconfirmed | Unconfirmed | Enterprise visibility monitoring |
| Semrush | Unconfirmed | Unconfirmed | SEO suite with AI tracking added |
| Ahrefs Brand Radar | Unconfirmed | Unconfirmed | Brand monitoring in AI search |
Treat "unconfirmed" as a prompt to ask, not as a no. Vendors ship quickly and I would rather tell you what I could not verify than guess.
Promptwatch: the one with documented IP verification
Promptwatch is the clearest fit if you want the crawler layer tied to everything else. Its Agent Analytics shows real-time, IP-verified logs for bots like GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Meta-WebIndexer, with the pages they read, the errors they hit and plain-language explanations of crawl failures. It then connects the crawl to a citation, so you see a citation rate per page rather than just a hit count.
The useful part is what sits around it. Visitor analytics show actual AI-referred traffic and conversions. Search Insights overlays Google Search Console and Bing data, so you can ask which pages rank in search but never get cited by AI. The Action Board and Content Agent then turn gaps into work. Most trackers stop at the dashboard, and this is the difference between knowing and fixing.
Logs can come in through Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint or the WordPress plugin. If you are on a locked-down enterprise stack, that list matters more than any feature page.
TreeHouse, the company behind Pararius, fixed crawler access and published proprietary rental data, and reports AI search at about 25% of organic conversions. That is the sort of outcome you only get after you can see what the bots see.
Pricing starts at $95 a month for Essential, which includes site audit and 360 Insights. Check promptwatch.com/pricing for current numbers, and note that crawler log volumes scale by plan.
The technical SEO route: JetOctopus, seoClarity and friends
If your team already does log file analysis, you may not need a GEO platform to get crawler data. You need your existing log tool to understand AI bots.
JetOctopus has a step-by-step guide on tracking AI crawler activity. It covers monitoring 40+ AI bots, verifying them against spoofing and joining logs with crawl data. It also sells a Log Analyzer and an AI Search Visibility product.
seoClarity lists Log File Analysis in its platform, alongside AI visibility tracking. I could not confirm whether that is IP-verified or AI-specific, so ask.
Other enterprise technical platforms worth putting on a shortlist and questioning:
The tradeoff is real. Log-first tools are strong on crawl behaviour and raw data but usually thinner on prompts, citations, sentiment and content workflows. GEO platforms are the reverse, with the exception of the few that do both.
Prompt trackers that mention crawler data
Some of the pure trackers are adding crawler features. One reviewer noted that Peec has built-in crawler log analysis plus MCP and API access.
I would not assume any of these verifies by IP. Ask, and ask for the answer in writing.
What the data says about why this matters
Promptwatch Data tracked weekly verified requests from June 8 to September 6, 2026. OpenAI accounted for 79.8% of verified AI crawler requests in the week of Aug 31 to Sep 6, down from 94.8% in the week of Jun 8 to 14. Share is moving, so your allow rules from last year may not cover this year's mix.
Meta is a good example. Promptwatch's Meta-WebIndexer data shows it growing from about 2% to nearly 38% of tracked AI crawler requests between mid-July and August 9, 2026. If your firewall treats unknown Meta agents as hostile, you may have cut yourself off from a fast-growing crawler without noticing.
The report's practical advice is simple. Check robots.txt, CDN and WAF rules for every provider in the chart. Then compare your own logs with the mix. If a provider is large in the benchmark and near zero in your logs, you are likely blocked or unreachable for it.
And crawling is only half the story. Promptwatch's content half-life data shows landing pages and docs stay cited longest while news lost half its pages within 36 days of the July 6, 2026 snapshot. Knowing which pages bots keep returning to tells you where to invest.
Questions to ask any vendor
These are the ones that separate a real crawler feature from a checkbox.
- How do you identify a bot: user agent, reverse DNS, or the operator's published IP ranges?
- Are spoofed requests excluded from the numbers, or just flagged?
- Which log sources do you ingest: Cloudflare, Fastly, Vercel, CloudFront, raw server logs?
- Do you separate training bots from search and user-triggered fetchers?
- Can I see crawl errors per page, with a reason I can act on?
- Do you connect crawls to citations and to AI-referred visits?
- What are the log volume limits on my plan, and what happens when I exceed them?
If a vendor cannot answer the first two in plain words, you are looking at tier 2 or 3.
How to choose
| Your situation | Sensible pick |
|---|---|
| You want crawler logs, citations, traffic and content fixes in one place | Promptwatch |
| You already run enterprise log analysis and want AI bots added | JetOctopus, Botify, Oncrawl or seoClarity, after asking about IP verification |
| You only need prompt monitoring on a budget | A tracker like Peec AI or Otterly.AI, and accept no verified crawler data |
| You are an agency managing many clients | A platform with white-label reporting and agency plans, such as Promptwatch's agency tiers |
If you want to browse more options, the GEO software directory lists dozens of platforms by category.
A quick DIY check before you buy
You can learn a lot for free. Pull a week of raw access logs, filter for AI user agents, and then check whether the source IPs fall in each operator's published ranges. Anything that does not is noise. Then look at status codes per bot. A pile of 403s for OAI-SearchBot is a finding worth more than any visibility score.
That manual process does not scale past a few sites, which is the real argument for a platform. But doing it once teaches you what good output looks like, and makes vendor demos much harder to bluff your way through.
Bottom line
IP-verified crawler logs are the least crowded part of AI visibility. Of the vendors I looked at, Promptwatch is the one with a documented, published verification method and the surrounding stack to act on it. Technical SEO platforms like JetOctopus are credible if logs are your home turf. Everyone else, ask the question and get the answer before you sign.