How to Crawl Documentation for RAG

Discovery

Start at /sitemap.xml. It is the cheapest discovery mechanism available and it usually reflects the structure the publisher intends. Treat it as a starting set rather than truth: sitemaps go stale, omit sections, and sometimes list URLs that redirect. Fall back to link-following inside the documented path scope for anything missing.

Crawl scope and URL rules

Write scope as explicit rules before running anything:

• Include only paths under the documentation root
• Exclude version archives unless you intend to index multiple versions
• Exclude non-primary locales, or index them with a locale field
• Exclude search result pages, tag indexes, and pagination shells
• Cap depth and total page count so a misconfiguration fails loudly
• Set a request rate that respects the site and any stated limits

Duplicate handling

Documentation sites generate duplicates prolifically: trailing-slash variants, locale mirrors, versioned copies of unchanged pages, and printer views. Respect the canonical tag, normalize URLs before comparison, and hash the normalized content to catch the rest. Without this, a 400-page set becomes several thousand near-identical chunks and retrieval precision collapses.

Page cleaning

Strip site chrome, sidebars, and version pickers. Protect code blocks, tables, and callouts explicitly — in technical documentation these are frequently the answer. Where the page uses tabs for language variants, either split them into separate documents or label each variant clearly; concatenating them creates documents that contradict themselves.

Chunking

Split on headings, not on a character count. A section is a coherent unit of meaning; 800 characters is not. Carry the page title and the full section path into each chunk so a retrieved fragment still knows where it came from, and avoid splitting in the middle of a procedure — a step without its prerequisites is worse than no step at all.

Document schema

{  "id": "docs/auth/api-keys#rotation",  "title": "Rotating API keys",  "section_path": ["Authentication", "API keys", "Rotation"],  "canonical_url": "https://example.com/docs/auth/api-keys",  "version": "v2",  "locale": "en",  "content_markdown": "## Rotation\nKeys can be rotated without downtime…",  "content_chars": 1840,  "code_blocks": 2,  "retrieved_at": "2026-09-14T09:41:00Z",  "content_hash": "sha256:…"}

Embeddings, conceptually

Embeddings turn each chunk into a vector so that semantically similar passages can be found. They locate related language; they do not verify correctness. Keep filterable metadata — version, product, section — alongside the vector, because narrowing the candidate set with a filter usually improves results more than tuning similarity.

Retrieval testing

Write down ten to twenty questions that the documentation should answer, ideally taken from real support tickets. Run them after every ingest and read what comes back. This test set will tell you about scope and cleaning problems faster than any aggregate metric.

Troubleshooting