Research note · Research note 05 — current edition
What AI Agents Need From the Open Web
Written by Aaron Grainger
Independent Content Strategist & Product-Marketing Writer · Published May 12, 2026
- Primary audience
- Product managers building AI features
- Also useful for
- AI engineers and technical founders
- Tone
- Authoritative
- Reading time
- 2 min
- Published
- May 12, 2026
Abstract
Agents interact with a web that was designed for human readers and search crawlers. This note sets out what an agent actually requires from a page, where current conventions fall short, and which of those gaps are a publisher's problem rather than an agent builder's.
Key questions
- What does an agent need that a human reader does not?
- Which existing web conventions already serve agents well?
- What should publishers do differently, and what should they not be asked to do?
How this note was developed
- — Inspection of pages from the perspective of a programmatic consumer.
- — Review of existing conventions — sitemaps, canonicals, structured data, feeds — against agent needs.
- — Independent perspective; no standards body position or consensus is implied.
Central arguments
01Agents need stable addressing more than rich presentation
A durable URL for a specific fact is worth more than an elegant page. Content that only exists behind interaction, infinite scroll, or a session-scoped view is effectively unaddressable and therefore uncitable.
02Change signals are the most underprovided thing on the web
Very few pages tell a consumer when their substantive content last changed. Last-modified headers are frequently the build time. This forces every monitoring system to reconstruct change detection from full content comparison.
03Existing conventions cover more than people assume
Sitemaps, canonical tags, feeds, and structured data already solve discovery, deduplication, and basic fact extraction where they are maintained. Much of the perceived gap is maintenance debt rather than missing standards.
04Access policy should be legible, not implicit
Publishers have legitimate reasons to restrict automated access. A clearly stated policy is better for both sides than silent blocking, which teaches pipelines to retry rather than to stop.
Limitations
- — This note takes an agent-builder's perspective and does not attempt to weigh publisher economics fully.
- — Conventions and norms in this area are actively changing.
Related reading
Related content
Field note · 3 min
Your AI Agent Does Not Need “the Internet.” It Needs a Retrieval Plan.
“Give the agent web access” is a capability statement pretending to be a requirement.
Research note · 2 min
The Web Context Problem
Most AI products that reason about the outside world fail for an unglamorous reason: the material they reason over is incomplete, stale, or structurally unusable. This note argues
Research note · 2 min
The Anatomy of a Source-Linked AI Answer
A source-linked answer is not an answer with links appended. This note breaks a well-formed answer into its parts — claim, support, provenance, confidence, and refusal — and argues
Research note · 2 min
Why Clean Content Beats Raw HTML for Most AI Tasks
Raw HTML is a rendering instruction set that happens to contain text. This note examines what is lost and gained when a page is normalized to structured Markdown, and identifies th
Version history
Current: 1.0 · Published
- 1.0May 12, 2026First published.
Was this useful?
Sourceframe is an independent product concept created for research, product-design, and technical-content exploration. It is not an operating company, and nothing here describes a live commercial service. All examples, schemas, and code are illustrative unless a page says otherwise. No client data, customer outcomes, performance results, or partnerships are described anywhere on this site.