Key takeaways
- Coding agents like Claude Code search, fetch docs, and pick a library on the developer's behalf, often without the developer ever opening your site.
- Most causes are mundane: blocked crawlers, JavaScript-only docs, no machine-readable overview, and no clear answer to "when should I use this over X?"
- Markdown barely matters for AI search (0.05% of citations) but matters for coding agents, which fetch specific docs and READMEs.
- Good docs for humans do not guarantee agent visibility. You have to test what the agent actually does.
- One caveat up front: there is no solid first-party study of Claude Code's recommendation bias. Treat the fixes below as a well-reasoned checklist, not proven law.
What is actually happening when Claude Code picks an SDK
A developer types "add authentication to my app" or "build a checkout flow." Claude Code does not just recall a library from memory. It can search, fetch docs and READMEs, compare options, and then write code against whichever one it settled on. The developer may never see your documentation.
That makes the agent a discovery channel. A Lightsage write-up on why Claude Code skips libraries describes the experience many devtool teams have had: a demo where the agent suggests the competitor, or a customer saying they found a rival because "Claude suggested it." Keep in mind that piece is a single vendor blog, and its claims are not backed by independent data. But the failure modes it lists match what the crawler data shows, so they are worth working through.

It also helps to understand the machinery. Anthropic describes the Claude Agent SDK as the harness behind Claude Code, built on one idea: Claude needs the same tools programmers use every day. Search, file reads, shell commands. Those tools are the path your product has to be visible on.
The good docs paradox
I think this is the most uncomfortable point for devtool teams. You can have beautiful docs, a clean API reference, and great tutorials, and still lose the recommendation. Humans scan docs. Agents fetch them, parse them, and try to run the result.
So the questions change:
- Can the agent's crawler reach your docs at all?
- Does the page render without client-side JavaScript?
- Is there a compact, machine-readable summary of what your API does?
- Can the agent actually execute code with your SDK and succeed?
- Is there a direct integration, such as an MCP server?
Your competitor with mediocre docs may simply win on some of those.
Reason 1: your docs are not crawlable
This is the cheapest one to check and the most common to find.
Promptwatch Data tracked Claude's citation crawler, Claude-User, from December 2025 to April 2026 (Claude citation crawler visits over time). It grew from about 30 visits a day in mid-December to several thousand a day by mid-April, more than 100x in four months. Claude is still only about 1% of tracked AI citation crawler traffic, with ChatGPT taking most of the rest. But the growth is real, and Claude-User may visit pages when someone asks Claude a question or uses Claude Code.
What to check:
- robots.txt rules that block ClaudeBot or Claude-User
- CDN or WAF bot rules that challenge or block unfamiliar user agents (very common with aggressive bot protection)
- Docs that only render through client-side JavaScript
- Server logs: do you see Anthropic user agents at all?
Promptwatch's advice from that report is to baseline your logs for Anthropic user agents and review them monthly. If you want a quick sanity check first, the AI crawler URL inspector and the robots.txt generator for AI are free.
Reason 2: you ignore markdown, where coding agents actually look
Here is a nuance that gets lost. For AI search in general, markdown is irrelevant: across ChatGPT, Claude, Perplexity and AI Overviews, HTML is 99.94% of citations and .md files are 0.05% (Markdown in AI search).
Coding agents behave differently. In the same report, Promptwatch saw a big increase in docs crawling by Claude's bot and OpenAI's bot, with /docs/**/*.md paths among the most requested (the top path had 3,241 requests). These agents fetch specific docs, READMEs and API references rather than searching the web like ChatGPT does. Markdown is cheaper to parse and keeps the context window clean.
The practical rules from that report:
- Do not build .md mirrors of your marketing pages. It will not help AI search.
- Do reserve .md endpoints for developer docs.
- Fix JavaScript-only rendering first.
A short llms.txt that points to your core docs is a reasonable companion. The LLMs.txt generator will draft one. I would not oversell it, since nobody has published proof that it moves recommendations, but it costs almost nothing.
Reason 3: the agent cannot tell when to pick you
Ask Claude directly: "When should I use [your SDK] versus [competitor]?" Read the answer like a skeptical buyer.
If the answer is vague or wrong, the agent has little to work with. Competitors who wrote explicit comparison pages and use-case docs give the model something concrete to repeat. You did not position yourself against them, so they got to position you.
Fixes that tend to help:
- A plain comparison page for each real competitor, written honestly (including where they win)
- Use-case pages phrased the way a developer would prompt: "add Stripe-style checkout to a Next.js app"
- Quickstarts that end with working code, not a link to another page
- Clear version numbers and deprecation notes, so the agent does not generate stale code
The ChatGPT Query Fan-Out Generator at promptwatch.com can help you brainstorm the sub-searches an AI might run for a given task.
Reason 4: you have no tool or MCP presence
If an agent can call your API through an MCP server or a packaged skill, it can verify the integration works. If it cannot, it recommends you abstractly and hands the developer code to run by hand. That is a worse experience, and plausibly makes the agent less likely to suggest you. This is the Lightsage claim, and it is logical rather than measured, so test it for your own product.
Minimum viable presence:
- An official MCP server with a few well-named tools for your top two or three tasks
- Listing in the public MCP registries
- A sandbox or test mode so the agent can try things without real credentials or money
- Errors that explain themselves in plain language, because the agent will read them
Reason 5: third-party sources tell a different story
The agent does not only read your site. For software prompts, Promptwatch Data on ChatGPT (Sep 28 to Oct 4, 2026) shows which sources get cited (software citations report). This is ChatGPT, not Claude Code, so use it as a proxy.
| Source type | Share of software citations |
|---|---|
| Docs and help centers | 2.64% |
| Vendor sites | 2.01% |
| Review sites and analysts | 1.57% |
| GitHub | 1.2% |
| App stores | 0.47% |
GitHub is the single most cited domain at 1.2%, then learn.microsoft.com at 0.98% and G2 at 0.73%. Vendor sites plus docs earn about 3x what review sites and analysts do. The takeaways are simple: keep your docs crawlable, keep a public and active GitHub presence, keep G2 and Capterra profiles current, and treat your own site as the main citation source.
A baseline test you can run this week
- Write 10 to 20 realistic developer tasks your product solves. Phrase them as prompts, not keywords.
- Run each in a clean Claude Code session with no project context. Record which library it picks, and whether it searched or fetched docs.
- Repeat with a couple of rephrasings, since results vary between runs.
- For every miss, note the reason: blocked, not found, found but not chosen, chosen but code failed.
- Fix the cheapest causes first (crawler access, rendering), then content, then integrations.
- Rerun monthly and compare.
This is the part people skip. Without the baseline you cannot tell whether a change worked.
Quick diagnostic table
| Symptom | Likely cause | First fix |
|---|---|---|
| Agent never mentions you | Crawlers blocked or JS-only docs | Allow Claude-User and ClaudeBot, server-render docs |
| Mentions you but picks a rival | Weak positioning content | Comparison and use-case pages |
| Picks you, generated code fails | Stale or ambiguous docs | Versioned quickstarts, working examples |
| Rival has an MCP server, you do not | No tool presence | Ship an MCP server for core tasks |
| Wrong claims about your pricing or features | Outdated third-party pages | Update G2, GitHub, and docs; publish a canonical facts page |
Tooling for tracking this
Most classic rank trackers will not tell you what Claude Code recommends. Tools that track AI answers and crawler activity are closer to the job. Promptwatch is an AI visibility platform that covers coding agent visibility (Codex, Claude Code and OpenCode recommendations), plus IP-verified AI crawler logs so you can separate an access problem from a content problem. Its AI crawler logs show which pages bots read and where they hit errors.

If you are comparing options, the GEO software directory lists others. Many are built for consumer brands, so check whether they track coding agents and not only chat assistants. Some alternatives from the catalog worth a look:

In my read, the monitoring-only tools tell you that you are missing, but not whether the agent could reach your docs in the first place. That second answer is usually where the fix lives.
Honest caveats
- Nobody has published a rigorous study of why Claude Code picks one SDK over another. The Promptwatch reports above cover crawler and docs-fetch behavior, not recommendation outcomes.
- Model behavior changes with every release. A fix that works this quarter may matter less next quarter.
- Claude Code also draws on training data. Crawl fixes help with live lookups; changing what a model "believes" takes longer and is partly out of your hands.
- Agents react to a developer's existing project. If a codebase already uses your competitor, no content fix will override that.
What to do first
Check robots.txt and your CDN bot rules today. Server-render your docs. Add a markdown endpoint for developer docs only. Write one honest comparison page for your biggest rival. Then run the baseline test and write down the numbers before you change anything else.
FAQ
My competitor has worse docs but gets recommended more. Why?
Usually one of three things: their docs are easier for an agent to fetch and parse, they published explicit comparison content, or they have an MCP server or other tool presence. Documentation quality for humans is only one input.
Does blocking AI crawlers hurt me with Claude Code?
It can. If Claude-User or ClaudeBot cannot fetch your docs, the agent has less to base a recommendation on. Review your robots.txt and WAF rules.
Should I convert my whole site to markdown?
No. Markdown is 0.05% of AI search citations. Keep it for developer docs, where coding agents do fetch it.
How often should I retest?
Monthly is a sensible cadence, and again after any major docs change or model release.

