Sourceframe

Platform model

From chaotic pages to usable AI context.

Sourceframe models a practical web-context pipeline for AI systems. This page describes the conceptual layers, what each is good for, and where each one breaks. It is a design model, not a purchasable service.

Layer 01

Source Discovery

Find relevant public sources for a question, task, or monitored topic, and keep the query that produced each candidate.

  1. 01Question
  2. 02Query set
  3. 03Candidates
  4. 04Selection

Real-world example

A research task on deployment regions expands into six phrasings, returning twelve candidate URLs with their originating query attached.

Design consideration

Discovery predicts relevance to a query, not the presence of a fact. A candidate set is a hypothesis; it never substitutes for retrieval.

Layer 02

Content Normalization

Transform cluttered pages into focused, readable content that keeps headings, lists, tables, code, and link targets.

  1. 01Raw page
  2. 02Strip chrome
  3. 03Protect structure
  4. 04Markdown

Real-world example

A documentation page of 180KB of markup reduces to 2KB of Markdown with both code blocks intact and the canonical URL retained.

Design consideration

Aggressive cleaning removes tables and code; conservative cleaning floods the index with navigation. Protected-element rules are required, and pages whose meaning is visual lose information no matter what.

Layer 03

Structured Extraction

Convert page information into schema-aligned records with types, optional fields, and a supporting excerpt for each value.

  1. 01Clean content
  2. 02Schema
  3. 03Extract
  4. 04Validate

Real-world example

A pricing page yields tier names, amounts, currency, and included seats — with nulls where the page says nothing.

Design consideration

Model-based extraction is tolerant of layout change and intolerant of ambiguity: it will resolve an unclear page into a confident value unless an excerpt is required.

Layer 04

Site Intelligence

Crawl related pages, documentation sets, directories, and content collections inside an explicitly defined scope.

  1. 01Root URL
  2. 02Scope rules
  3. 03Crawl
  4. 04Dedupe

Real-world example

A documentation root plus path rules produces 412 canonical pages, with locale mirrors and version archives excluded.

Design consideration

Crawl problems are scope problems. Too wide and duplicates dominate the index; too narrow and the page that mattered was never fetched.

Layer 05

Source Traceability

Keep the original URL, retrieval time, and supporting excerpt connected to every downstream output.

  1. 01Retrieval record
  2. 02Claim
  3. 03Excerpt
  4. 04Cited output

Real-world example

An answer sentence carries a claim id, which carries a URL, a timestamp, and the sentence from the page that supports it.

Design consideration

Provenance can guarantee where a claim came from. It cannot guarantee the source is correct, and presenting it as certainty transfers the open web's errors onto your product.

Discovery predicts relevance to a query, not the presence of a fact. A candidate set is a hypothesis, never a substitute for retrieval.
Working principle — source discovery

Conceptual API flow

Illustrative workflow pseudocode

The snippets below describe the shape of a web-context call. They are illustrative only — there is no Sourceframe API to run them against, and nothing here should be read as a description of a live service.

TypeScript — illustrative workflow pseudocode
// 1. Input URL and requested output formatconst source = await sourceframe.retrieve({  url: "https://example.com",  format: "markdown",  includeSourceMetadata: true   // 2. carry provenance from the first step}); // 3. Optional extraction schema over the same contentconst record = await sourceframe.extract({  content: source.markdown,  schema: {    title: "string",    summary: "string",    key_facts: ["string"]  },  requireExcerpt: true,          // 4. every value quotes its supporting text  allowNull: true                //    "not found" is a valid answer}); // 5. AI-ready result object// { content, canonical_url, retrieved_at, record, excerpts }
Python — illustrative workflow pseudocode
# Illustrative only — not a runnable API.result = sourceframe.extract(    url="https://example.com",    schema={        "title": "string",        "summary": "string",        "key_facts": ["string"]    }) # Source metadata travels with the record, not bolted on afterwardsprint(result.source_url, result.retrieved_at)print(result.missing)   # fields the page did not establish

Why this layer matters

Five reasons teams end up building this layer

  1. 01

    AI systems need current context

    Model knowledge is frozen; the questions people ask are not.

  2. 02

    Raw HTML is rarely usable as-is

    Most of a page is rendering instruction, and boilerplate competes with content during retrieval.

  3. 03

    Retrieval quality shapes output quality

    A better model describes bad context more convincingly; it does not correct it.

  4. 04

    Structured extraction reduces downstream ambiguity

    Typed records with explicit nulls turn silent wrongness into visible failure.

  5. 05

    Source links make research accountable

    Provenance is what lets a product separate what it knows from what it inferred.

See these layers used in workflow patterns →