Key takeaways
- Coding agents like Claude Code are a discovery channel for developer tools. A developer asks for "auth" or "a checkout flow" and the agent picks an SDK, often without the developer opening your docs.
- Good docs don't guarantee a recommendation. Agents need to reach your docs, understand what your API is for, and succeed when they write code against it.
- Four causes show up repeatedly: blocked or unrenderable docs, no machine-readable overview, a vague API surface, and no MCP or tool presence. Each has a five-minute check.
- Claude's citation crawler (Claude-User) grew more than 100x in four months, from about 30 visits a day in December 2025 to several thousand by mid-April 2026, according to Promptwatch Data. It is still about 1% of tracked crawler traffic, so treat it as a fast-growing secondary channel.
- Measure before you fix. Run the same task prompts monthly, log what gets recommended, and watch your crawler logs.
What is actually happening
A developer types "add authentication to my app" into Claude Code. The agent searches, fetches a few pages, weighs options, and writes code. The developer reviews the diff. At no point did anyone visit your pricing page or read your carefully written quickstart.
If that flow ends with your competitor's package in package.json, you lost a deal you never knew existed. The usual signal is anecdotal: a customer mentions they found the competing tool because "Claude suggested it." By then it has happened hundreds of times.
I want to be upfront about the evidence here. Most of the specific advice about why Claude Code skips a library comes from practitioner write-ups, notably Lightsage's analysis, and Lightsage sells an agent-led growth product, so read it as informed vendor opinion. I couldn't find first-party data that measures which SDKs Claude Code recommends or that quantifies the effect of llms.txt or MCP servers. What follows is a reasoned checklist, not a proven formula. Test it against your own results.

The good docs paradox
The most counterintuitive claim in the Lightsage piece is also the one I find most plausible: great documentation does not equal AI visibility. An API with Stripe-quality tutorials can get no recommendations while a competitor with mediocre docs dominates.
Humans and agents read differently. A human skims navigation, trusts a brand, and tolerates a sidebar. An agent fetches a handful of URLs, extracts text, and moves on. It cares about:
- whether it can fetch and parse your pages at all
- whether there is a compact, machine-readable description of what you do
- whether code written against your API actually runs
- whether it can call you directly through a tool integration
Notice that none of those are about prose quality. That is why a competitor with worse writing can win.
Four reasons and quick checks
| Problem | Quick check | Fix |
|---|---|---|
| Docs aren't crawlable | Look in robots.txt and CDN/WAF rules for blocks on ClaudeBot, Claude-User, GPTBot | Allow AI crawlers, server-render docs |
| No llms.txt | Open yourdomain.com/llms.txt | Publish a concise API overview |
| Ambiguous API surface | Ask Claude "when should I use [your SDK]?" | Add use-case and comparison pages |
| No MCP or tool presence | Search MCP registries for your product | Ship an MCP server or Claude skill |
Reason 1: your docs can't be reached
This is the boring one and it is the one I would check first. Plenty of docs sites sit behind bot protection that treats every unfamiliar user agent as hostile. Others render everything client-side, so a fetch returns an empty shell.
Promptwatch's own recommendation from its Claude crawler report is direct: confirm that Claude-User and ClaudeBot aren't blocked in robots.txt or by CDN and WAF bot rules, then search your server or CDN logs for Anthropic user agents to set a baseline and review it monthly. Claude-User is the crawler that visits when someone asks Claude a question or uses Claude Code.
A minimal robots.txt that doesn't get in the way looks like this:
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
Then fix rendering. Promptwatch's markdown report says to deal with heavy client-side JavaScript first. If the HTML you serve doesn't contain the docs text, nothing else on this list matters.
If you want to see which AI bots actually reach your docs and where they hit errors, tools like Promptwatch have IP-verified AI crawler logs that separate real Anthropic and OpenAI traffic from spoofed bots. The free AI Crawler URL Inspector is a quicker first pass on a single URL.

Reason 2: no machine-readable overview
An llms.txt file is a plain-text index that tells an agent what your product is and where the important pages live. Adoption is uneven, and I haven't seen hard data that it changes recommendations, so don't oversell it internally. It is cheap, though. An hour of work, and it forces you to write the one-paragraph description of your API that most teams never get around to.
What to put in it:
- one or two sentences on what the SDK does and who it is for
- links to the quickstart, API reference, and auth docs
- the languages and frameworks you support
- when not to use you (this builds more trust than it costs)
Promptwatch has a free LLMs.txt generator if you want a starting draft.
Reason 3: the API surface is ambiguous
Ask Claude "when should I use [your SDK]?" and read the answer closely. If it describes you vaguely, mixes you up with a neighbour, or lists use cases you don't serve, the agent has the same fuzzy picture when a developer asks for a recommendation.
Agents lean on comparison and decision content. If the only pages that explain your product are marketing pages, an agent has nothing concrete to cite when it weighs you against a rival. The fix is plain content work:
- a "when to use us, when to use X" page for each major competitor
- task-shaped docs: "add passwordless login in Next.js", not just "Authentication overview"
- a clear page on limits, pricing model, and supported platforms
This also matches what Promptwatch Data shows for ChatGPT on software prompts (Sep 28 to Oct 4, 2026). Docs and help centers earned 2.64% of software citations and vendor sites 2.01%, together about three times the 1.57% for review sites and analysts. That is ChatGPT, not Claude Code, so it is a proxy. Still, it says your own pages can carry real weight if they are crawlable and specific.
Reason 4: no MCP or tool presence
The Lightsage argument is that an API with MCP presence can be used directly by the agent, while one without it gets recommended abstractly, leaving the developer to run generated code by hand. In their FAQ, a competitor with worse docs wins partly because agents prefer APIs they can interact with directly. That claim is unquantified, so treat it as a hypothesis. But it fits how Claude Code works: the agent can call tools, and a tool that works in the session is easier to keep using than one that needs a manual setup step.
The practical steps:
- Search public MCP registries for your product and your competitors. Note who is listed.
- If you aren't there, build a small MCP server covering the three or four tasks developers most often start with (create a project, send a first request, check status).
- Consider a Claude skill that packages your best-practice usage so the agent follows your patterns.
- Document installation in one copy-paste command.
Keep the first version small. A narrow server that works beats a sprawling one that half works, because the agent's experience of failure is what shapes later behaviour.
Docs should be easy for agents to fetch
Coding agents don't search the web the way a chat assistant does. Promptwatch's markdown report describes them fetching specific docs, READMEs, and API references. Its crawl log shows paths like /docs/.../*.md being requested by Anthropic's ClaudeBot and OpenAI's search bot, with the top path at 3,241 requests.
That doesn't mean markdown helps you rank in AI search. The same report found HTML is 99.94% of AI search citations and markdown only 0.05%. Its conclusion: markdown serves agents, not AI search. So the sensible split is:
- keep normal HTML pages for search and citations
- offer .md versions of developer docs, where agents fetch them directly because they are cleaner and cheaper to parse
- don't bother mirroring your marketing pages as markdown
Agent-native search engines such as Exa, already built into tools like OpenCode, let agents find and retrieve URLs on their own. Another reason to make sure your docs are indexable and not hidden behind a login or JS wall.
Keep a public GitHub presence
In the software citation data above, github.com was the single most cited domain at 1.2% of ChatGPT software citations, ahead of Microsoft's docs at 0.98% and G2 at 0.73%. A well-kept repo with a clear README, examples, and active issues is another surface an agent can read. Review sites matter too: G2 and Capterra both appear in the top citations, so keep those profiles accurate.
| Source in ChatGPT software citations | Share |
|---|---|
| github.com | 1.2% |
| learn.microsoft.com | 0.98% |
| g2.com | 0.73% |
| docs.aws.amazon.com | 0.41% |
| gartner.com | 0.34% |
| capterra.com | 0.32% |
Period: Sep 28 to Oct 4, 2026, from Promptwatch Data. One week is a thin sample. Check the report for the current picture.
A 30-day plan to close the gap
Week 1: baseline
Write 10 to 15 prompts that mirror the tasks your customers do. Not "what is the best SDK" but "add webhooks to my Express app" or "set up usage-based billing". Run each in Claude Code (and Codex or Cursor if your users use them), in a clean project, several times. Record which tool is picked, which pages the agent fetched, and whether the generated code ran.
The point is to separate three failure types: not found, found but not recommended, and recommended but the code failed. They need different fixes.
Week 2: remove access problems
Fix robots.txt and WAF rules, server-render the docs, and check your logs for Anthropic and OpenAI user agents. Confirm the crawl failures drop. Remember Claude is still around 1% of tracked citation-crawler traffic and ChatGPT dominates the rest, so the same fixes help you in both.
Week 3: clarify the surface
Publish llms.txt, add the comparison and use-case pages, and rewrite the top quickstart so it ends in a working example the agent can copy. Add a short "limitations" section. Honest limits help an agent match you to the right job.
Week 4: ship tool presence
Release a minimal MCP server or skill, list it in registries, and re-run your baseline prompts. Compare.
Measuring whether it worked
Adjust expectations: you will not get a clean rank number like in classic SEO. Recommendations vary run to run. Track the share of runs where you are picked, the share where code succeeds, and the crawl rate on your docs.
For the monitoring side, a few categories of tools help:
| Need | Option |
|---|---|
| Track how AI engines and coding agents (Codex, Claude Code, OpenCode) mention and cite you, plus crawler logs | Promptwatch |
| Enterprise AI visibility monitoring | [tool:profound] |
| Lightweight prompt tracking | [tool:otterly-ai] |
| Free brand visibility checks | [tool:rightchoice-ai] |
Promptwatch is the one in that list that tracks coding agent recommendations directly, alongside crawler logs and AI-referred traffic, which is why it fits developer-tool teams. If you want to compare more options, the directory at bestgeosoftware.com lists GEO platforms side by side.
One honest caveat. Tracking tools measure prompts you choose. They can't tell you what a developer typed into a private terminal session, so combine them with a simple "how did you hear about us" field at signup and a look at your package download trends.
What I would not spend time on
- Chasing a perfect llms.txt before fixing crawl access. Access first.
- Mirroring every page as markdown. Reserve it for docs.
- Rewriting docs for style. The paradox is real: polish rarely moves the needle here.
- Assuming one agent equals all agents. Test in more than one.
Final checklist
- robots.txt and CDN rules allow ClaudeBot, Claude-User, GPTBot, and OAI-SearchBot.
- Docs return real text without JavaScript.
- llms.txt exists and says what you are and aren't for.
- Comparison and task-shaped pages exist for your top three rivals.
- A small MCP server or skill is published and listed.
- A public repo with a clear README and working examples.
- A monthly run of your baseline prompts, logged.
If you do only two things this week, check the logs for Anthropic user agents and ask Claude when it would use your SDK. Both take minutes, and the answers usually tell you which of the four problems you have.