Sourceframe

Workflow pattern / Knowledge base

Create a web-to-knowledge-base pipeline

A general ingest pipeline that turns an approved source list into a maintained, queryable knowledge base.

Advanced / Markdown / Continuous

The problem

Teams build one-off ingests per source, each with its own quirks, and none of them are maintained after launch.

Who it is for

Platform teams standardising how external content enters internal systems.

Teams: Platform / Data / Support

Inputs

  • Approved source registry
  • Per-source scope rules
  • Document schema
  • Refresh policy

Conceptual process

  1. 01Register

    Every source is registered with scope, owner, cadence, and access policy.

  2. 02Ingest

    One shared crawl-and-normalize path, configured per source rather than reimplemented.

  3. 03Standardise

    Emit the same document schema regardless of source type.

  4. 04Index

    Chunk on structure and write with delete-on-removal semantics.

  5. 05Observe

    Track per-source document counts and content volume to catch silent breakage.

  6. 06Refresh

    Recrawl on cadence and re-run the retrieval regression set.

Flow diagram

  1. 01Source registry
  2. 02Crawl
  3. 03Normalize
  4. 04Standard schema
  5. 05Chunk + index
  6. 06Run report

Example output

Illustrative output
{  "source_id": "vendor-docs",  "documents": 412,  "documents_previous_run": 418,  "delta_reason_required": true,  "median_content_chars": 2840,  "last_run": "2026-09-14T02:00:00Z",  "failures": [{ "url": "https://example.com/docs/legacy", "reason": "empty after normalization" }]}

Data-quality considerations

  • A drop in document count or median content length is a defect signal, not a content signal.
  • Every document carries source, canonical URL, and retrieval time.
  • Sources without an owner get removed, not carried indefinitely.

Failure modes

  • Per-source custom code that nobody maintains after the author leaves.
  • Index grows forever because removals are never processed.
  • One broken source degrades the whole knowledge base without alerting.

Suggested architecture

  • Source registry
  • Shared crawl/normalize service
  • Schema writer
  • Structure-aware chunker
  • Index with deletes
  • Run observability

What to test first

  1. 01Onboard a second source and confirm no new bespoke code was required.
  2. 02Break one source and confirm the run report flags it.

Related reading