Search and source discovery
Finding the public pages that could plausibly answer a question — and knowing when none do.
Web context systems for practical AI
The web is not a database. AI systems keep treating it like one.
Sourceframe is an independent exploration of what sits between a URL and a usable fact: discovery, retrieval, normalization, extraction, and the provenance record that lets a product say where an answer came from. Written with the trade-offs left in.
- 01URL
- 02Discovery
- 03Retrieval
- 04Normalization
- 05Structured output
- 06AI workflow
- 07Cited result
A conceptual sequence, not a running service. Each stage is separately inspectable, and each one can fail in its own way.
01
Current sources
Context is only useful if it reflects the page as it exists now.
02
Structured outputs
Defined shapes beat free text when software has to consume the result.
03
Traceable context
Every claim keeps a path back to the URL it came from.
04
Production-minded workflows
Designs that assume failure, drift, and rate limits from the start.
What Sourceframe explores
Six problems that show up in every web-enabled AI build
Each one is cheap to ignore during a prototype and expensive to discover in production.
Website-to-Markdown transformation
Reducing a rendered page to the content a model can actually use.
Schema-driven extraction
Turning inconsistent pages into typed records with explicit nulls.
Documentation crawling for RAG
Scope rules, deduplication, and chunking that survives a recrawl.
Web-change monitoring
Separating changes that matter from the churn that does not.
Cited AI research workflows
Answers where every claim keeps a path back to its source.
How it works
Six stages, each separately inspectable
- Stage 1
Identify relevant sources
- Stage 2
Retrieve the useful content
- Stage 3
Normalize page noise into usable context
- Stage 4
Extract facts into defined structures
- Stage 5
Deliver context to an AI application
- Stage 6
Preserve source-level traceability
Provenance can tell you where a claim came from. It cannot tell you the claim is true — and presenting it as certainty moves the open web's errors into your product.
Featured field guides
Start here
AI Agents
How AI Agents Browse the Web Reliably
A practical guide to discovery, retrieval, clean context, structured extraction, and source traceability.
Intermediate6 min read
Foundations
How to Turn a Website Into LLM-Ready Markdown
Why clean, structured text often matters more than raw HTML in AI workflows.
Introductory4 min read
Decision Guides
Web Scraping vs Crawling vs Search for AI Systems
Choose the right web-data approach for research, RAG, monitoring, and agent workflows.
Introductory4 min read
Research Workflows
How to Build a Cited AI Research Agent
A source-first workflow for turning open-web information into accountable AI answers.
Advanced4 min read
RAG and Knowledge Systems
How to Crawl Documentation for RAG
Turn product documentation into a useful, maintainable knowledge source.
Intermediate4 min read
Web Monitoring
How to Monitor Web Pages for Meaningful Changes
Separate signal from noise when product pages, policies, prices, and documentation update.
Intermediate4 min read
Featured workflow patterns
Patterns with failure modes attached
Knowledge base
Turn a documentation site into a RAG knowledge base
Crawl a documentation set, clean it, chunk it on structure, and keep it current without rebuilding from scratch.
IntermediateMarkdownWeekly
Research
Build a cited AI research assistant
Answer open questions from current public sources, with every claim linked to the passage that supports it.
AdvancedAnswer + citationsOn demand
Competitive intelligence
Build a content-intelligence workflow for editorial teams
Understand how a topic is being covered publicly, where the gaps are, and which formats recur.
IntermediateDatasetWeekly
Monitoring
Monitor competitor pages for meaningful changes
Watch a defined set of public pages and alert only when a change could plausibly change a decision.
IntermediateDigestDaily
Research
Create a vendor-comparison assistant
Compare vendors on declared criteria using each vendor's own public pages, with disagreement preserved.
AdvancedAnswer + citationsOn demand
Research note
The Web Context Problem: Why AI products fail without usable source data
Most AI products that reason about the outside world fail for an unglamorous reason: the material they reason over is incomplete, stale, or structurally unusable. This note argues that web context is a distinct infrastructure layer with its own quality properties, and that treating it as a solved detail of prompt engin…
Read the research note →- Edition
- Research note 01 — current edition
- Key questions
- 4 examined
- Status
- Independent analysis — no external validation claimed
Experiment log
What's currently being tested
| Experiment | Status | Next step |
|---|---|---|
| Direct-answer-first guide structure | Published as a design exploration | Applied across guides 1–10. Next step: test whether the takeaway list … |
| Comparison-table placement in technical content | In synthesis | Early impressions favour table-early when options are parallel, table-… |
| Tutorial code sample position | Prototype reviewed | Next step: try a toggle between 'whole sample' and 'step by step' on o… |
| Decision-tree content format | Awaiting user testing | Live in the Tools section. Needs at least five external reviewers.… |