Sourceframe

Workflow pattern / Knowledge base

Turn a documentation site into a RAG knowledge base

Crawl a documentation set, clean it, chunk it on structure, and keep it current without rebuilding from scratch.

Intermediate / Markdown / Weekly

The problem

Support and in-product assistants answer from stale or partial documentation because nobody owns the ingest pipeline.

Who it is for

Platform teams building assistants over their own or a vendor's docs.

Teams: Platform / Documentation / Support

Inputs

  • Documentation root URL
  • Sitemap where available
  • Version and locale rules
  • Exclusion path list

Conceptual process

  1. 01Discover

    Start from the sitemap, fall back to link following within the documented path scope.

  2. 02Scope

    Apply include/exclude rules for versions, locales, and changelog archives.

  3. 03Deduplicate

    Respect canonical tags and collapse near-identical version pages.

  4. 04Clean

    Strip navigation and chrome while explicitly protecting code blocks and tables.

  5. 05Chunk

    Split on headings; carry title and section path into every chunk.

  6. 06Test

    Run a fixed retrieval question set and review what comes back.

  7. 07Maintain

    Recrawl on a cadence, delete removed pages, and record last-retrieved per document.

Flow diagram

  1. 01Sitemap
  2. 02Scoped crawl
  3. 03Dedupe
  4. 04Normalize
  5. 05Chunk
  6. 06Index
  7. 07Retrieval test

Example output

Illustrative output
{  "id": "docs/auth/api-keys#rotation",  "title": "Rotating API keys",  "section_path": ["Authentication", "API keys", "Rotation"],  "canonical_url": "https://example.com/docs/auth/api-keys",  "version": "v2",  "content": "## Rotation\nKeys can be rotated without downtime by ...",  "retrieved_at": "2026-09-14T09:41:00Z"}

Data-quality considerations

  • One H1 per page; if a template repeats the site name as H1, fix it in normalization.
  • Never index pages that are pure navigation.
  • Keep version as a filterable field rather than mixing versions in one index.

Failure modes

  • Locale duplicates inflate the index and dilute retrieval.
  • Code blocks stripped by aggressive readability extraction.
  • Deleted pages remain retrievable months after removal.
  • Chunks split mid-procedure, so a retrieved step loses its prerequisites.

Suggested architecture

  • Sitemap reader plus scoped crawler
  • Canonical-aware dedupe stage
  • Normalizer with protected-element rules
  • Structure-aware chunker
  • Index writer with delete-on-removal
  • Scheduled recrawl with a retrieval regression set

What to test first

  1. 01Retrieve answers for the ten questions support actually receives.
  2. 02Confirm a deliberately removed page disappears after one recrawl.
  3. 03Check that a code-heavy page keeps every code block.

Related reading