Your Docs Aren't Agent-Ready Just Because Bots Can Crawl Them

Passing a crawler check is the easiest part of agent-ready documentation.

An AI agent can fetch your page and still give the wrong answer. It can retrieve an old article, miss a permission requirement, combine two conflicting procedures, or cite a page that barely supports its claim.

A person can scan the navigation, open four tabs, notice that one article is dated, and recover from a vague heading. An agent often sees a search result, one Markdown page, or a handful of retrieved passages. Ambiguity that feels mildly annoying to a person can become a confidently wrong answer.

Call your documentation agent-ready only when an authorized agent can:

1. Find the correct source.
2. Retrieve a clean, current representation.
3. Understand the page's purpose.
4. Extract the full answer, including constraints.
5. Reject stale or contradictory guidance.
6. Cite the canonical page a person can verify.

llms.txt, crawler rules, and Markdown endpoints help with the first two steps. They do not prove the other four.

The standard agent-readiness audit stops too early

Cloudflare's 2026 scan of 200,000 popular domains found that 78% exposed robots.txt, while 4% declared Content Signals and 3.9% supported Markdown content negotiation. Fewer than 15 sites in the dataset exposed an MCP Server Card or API Catalog. Those figures show how early the transport layer still is. They do not tell us whether any retrieved page produced a correct answer. See Cloudflare's agent-readiness research.

Documentation teams also need to separate three related jobs:

• Search readiness covers crawling, indexing, internal links, and page quality for search engines.
• Agent retrieval readiness covers direct access to a useful representation within a practical context budget.
• Answer readiness covers whether the retrieved material contains enough current evidence to answer a real question safely.

Google makes the boundary especially clear. Its AI search features use the same crawlability and people-first content foundations as the rest of Google Search. Google says llms.txt, Markdown, and other AI-specific files neither help nor hurt visibility in its search results. Read Google's guidance for AI features and its generative AI optimization guide.

Direct agents and retrieval systems may still benefit from Markdown or a concise documentation index. Build those surfaces for the systems that use them. Do not sell them internally as a hidden Google ranking switch.

We have seen the same separation in Userorbit's own measurement. During one three-month observation window, Bing reported about 1,500 AI citations for our pages, with four pages accounting for 970. Citation coverage rose faster than clean referral and conversion signals.

That internal sample is not a benchmark. It is a warning against measuring one convenient step in a longer chain:

availability -> retrieval -> answer quality -> citation -> visit -> useful outcome

A large citation count can coexist with weak answers or little product impact.

Use three layers: retrievable, answerable, and proven

An audit becomes useful when it distinguishes technical availability from answer quality and observed outcomes.

A site can pass every header check and fail the second layer. It can also publish beautifully structured articles and fail the third because an obsolete URL keeps outranking the replacement.

A practical 100-point score

Use the score to locate weak areas, not to manufacture a universal grade.

Add gates before you publish the score:

• State how many pages, locales, and representations were tested.
• Remove genuinely inapplicable checks from the denominator instead of marking them as failures.
• Label model-assisted judgments and show the passages behind them.
• Reserve “proven” for audits that ran an outcome evaluation.
• Cap the maturity level when you find a critical privacy, access, canonical, contradiction, or deprecation failure.

Without those gates, two scanners can give the same documentation site very different scores and both appear correct. Their scopes, test presets, and definitions may simply differ.

Layer 1: Make the current source retrievable

The first layer is technical, but it still requires product decisions. A public help article, an authenticated API reference, and a customer-specific support record should not share one access policy.

Give each crawler an intentional policy

Crawler names describe different uses. OpenAI distinguishes OAI-SearchBot from GPTBot, while Anthropic separates ClaudeBot, Claude-User, and Claude-SearchBot.

Decide which uses your organization permits:

• search discovery and citations
• retrieval initiated by a user
• model training
• access to authenticated or licensed material

Then encode the policy with the mechanisms each provider supports. Keep the decision somewhere your security, legal, and documentation teams can review when crawler behavior or names change.

Protected documentation can still be agent-ready. It needs a deliberate path with scoped credentials and clear retention behavior. Publishing a hidden Markdown route that bypasses authorization is an exposure, not an optimization.

Make every representation resolve to one identity

Your sitemap should contain canonical public pages and honest lastmod values. A deployment timestamp does not become a meaningful freshness signal merely because the HTML changed.

HTML, .md routes, localized routes, and negotiated Markdown may represent the same article. Make that relationship explicit and avoid redirect chains or competing canonicals.

When a page becomes obsolete, do more than redirect it:

• label the old behavior
• name the replacement
• preserve migration context when readers still need it
• update internal links and machine-readable indexes

Old URLs remain in bookmarks, search results, caches, and previous model context. Deprecation belongs in answer safety.

Serve the article, not the website chrome

A documentation URL can serve HTML to a browser and Markdown to an agent:

GET /help/article
Accept: text/markdown
 
HTTP/2 200
Content-Type: text/markdown; charset=utf-8
Vary: Accept

Vary: Accept keeps shared caches from returning Markdown to a browser or HTML to an agent.

The machine-readable version should preserve the title, orientation, headings, code, warnings, tables, essential links, and canonical source. Remove navigation, cookie copy, empty controls, and repeated footer text. If the answer begins after thousands of tokens of chrome, the page is reachable but wasteful.

Tabs, accordions, diagrams, videos, and embedded demos need text equivalents. Hidden tab content and visual warnings should not disappear from the agent representation.

For the implementation details, read our companion guide to optimizing help center content for AI agents. That article owns the Markdown delivery pattern; this framework owns the audit.

Treat llms.txt as a map

The llms.txt proposal describes a compact Markdown index for important resources. A useful index has a clear summary, meaningful sections, stable links, and descriptions that help an agent choose the next page.

A flat export of every URL pushes the discovery problem onto the agent. Large corpora may work better with focused indexes such as /help/llms.txt, /developers/llms.txt, and /api/llms.txt, connected by a short root file.

Measure file size, duplicate content, update reliability, and retrieval performance before adding llms-full.txt. More context can produce more noise.

Layer 2: Make each page answerable

Clean Markdown cannot repair a vague article. The source itself needs an answer shape that matches the reader's job.

We use Diátaxis to separate tutorials, how-to guides, reference, and explanation. FAQ and troubleshooting pages deserve their own explicit patterns because support questions often arrive in those forms.

Mixing every job into one long page makes retrieval harder. A setup guide does not need a full architecture essay between steps two and three. Link to the explanation instead.

Put constraints beside the instruction

Agents often lose requirements that live far from the step they modify. Keep material constraints close to the action:

• required role or permission
• plan or feature availability
• starting state
• default behavior
• destructive or irreversible effects
• expected output

Verification is part of the answer. “Click Save” explains an action. “The status changes to Active and a test request returns 200” explains whether it worked.

Recovery deserves the same precision. Connect each fix to an observable symptom and state what evidence a user should collect before escalation.

Make trust visible across the corpus

An agent comparing several candidate pages needs evidence that one is current and authoritative. Track an accountable owner, last meaningful review, applicable product version, and verification source.

Then run corpus-level checks for contradictions:

• different defaults for the same setting
• old and new product names presented as current
• conflicting plan or permission requirements
• duplicate procedures with diverging steps
• how-to guides that disagree with reference pages

One correct page does not cancel a wrong page when retrieval can select either.

Coverage belongs in the same layer. Compare documentation with zero-result help searches, repeated support tickets, onboarding questions, and the product capability inventory. An agent cannot cite an answer nobody wrote.

Layer 3: Test real questions, answers, and citations

Outcome testing separates a plausible documentation system from a proven one.

Start with the questions users already ask

Build a small set from help searches, support conversations, onboarding sessions, and agent interactions. Remove sensitive data but preserve the user's wording.

Include several answer shapes:

• a single fact
• a multi-step task
• a permission or plan constraint
• a troubleshooting case
• a comparison
• deprecated behavior
• an ambiguous product term

Synthetic questions can fill gaps, but label them and have a domain expert review the reference answer.

Record gold evidence before running the model

For every question, define:

• the canonical source or source set
• a reference answer
• facts that must appear
• outdated or disallowed claims
• prerequisites and exceptions that must survive compression

This keeps grading anchored to evidence. Otherwise, one model writes the answer and another model grades whether the prose sounds convincing.

Measure retrieval before rewriting prompts

Use retrieval metrics that reveal where the source appeared:

• Recall at k: did the candidate set include enough evidence?
• Reciprocal rank: how early did the best source appear?
• nDCG: were the more useful pages ranked ahead of weaker ones?

If retrieval selects the wrong page, a more elaborate answer prompt will not repair the corpus.

Then grade the answer for correctness, completeness, groundedness, preserved constraints, task-step coverage, and deprecation awareness. Show the evidence passages behind each grade.

Measure citations separately. A citation should support the adjacent claim, point to the current canonical source, and remain accessible to the reader. Track the canonical citation rate, not just the total number of citations.

Finally, record the corpus version, question-set version, model and date, retrieval settings, prompts, and grading rubric. Models change. Without a reproducible setup, a model update can look like a documentation improvement.

Run the audit on a real slice of the corpus

Do not begin with every page. Choose one product area, one locale, and 20 to 50 questions that represent real work.

For each audit dimension, save evidence rather than a pass/fail label.

The score should make the weakest layer obvious. The evidence should make the next edit obvious.

Fix failures in risk order

Use this sequence:

1. Access and privacy failures. Repair accidental blocks, exposed private content, and broken authorization paths.
2. Wrong or deprecated guidance. Correctness outranks format.
3. Missing task-critical answers. Document repeated questions before polishing low-demand pages.
4. Retrieval failures. Repair sitemap coverage, canonical identity, internal links, and representation parity.
5. Incomplete answer shapes. Add prerequisites, defaults, verification, and recovery.
6. Retrieval cost. Remove chrome, split oversized pages, and improve indexes.
7. Emerging protocols. Add MCP, Agent Skills, API discovery, or commerce protocols when the product and audience have a real use for them.

That order keeps an agent-readiness project focused on correct answers instead of standards collecting.

Where Userorbit fits

Userorbit help centers expose the same article as a browser page or clean Markdown. Teams can provide a focused llms.txt index, a full Markdown export when appropriate, stable .md URLs, and negotiated Markdown through Accept: text/markdown.

Those retrieval surfaces solve one layer of the framework. The underlying documentation still needs clear jobs, current constraints, consistent terminology, and outcome testing.

That is why we are building a documentation-specific agent-readiness scanner around evidence rather than file counting. It will test retrieval, answer structure, freshness, contradictions, and citation behavior against a disclosed sample.

Start with one representative question

Pick a task customers regularly get wrong. Write down the canonical source, the facts a good answer must include, and the outdated claims it must avoid. Run the question through your current retrieval path and inspect every passage the agent used.

That single trace will tell you more than a generic badge. It exposes whether the agent found the right page, preserved the constraints, and cited something a person can verify.

Repeat the test across real questions and versions. Documentation becomes agent-ready when that chain holds consistently, not when a scanner finds one more file.