Documentation · Documentation version 1.0

Structured Extraction Patterns

A reference for schema design, validation, retries, review thresholds, and error handling in page-to-record extraction.

Independent Content Strategist & Product-Marketing Writer · Published Apr 28, 2025

Primary audience: AI engineers and technical founders
Also useful for: Data and operations teams using public-web information
Tone: Authoritative
Reading time: 4 min
Published: Apr 28, 2025

In short

Structured extraction converts page content into typed records against a schema you define in advance. The patterns that decide whether it holds up in production are schema design, explicit nulls, excerpt requirements, validation before use, and a defined review path for low-confidence records. Everything here is conceptual and illustrative; none of it describes a live service.

On this page

When to use structured extraction
Input design
Schema design principles
Validation patterns
Retry and fallback
Confidence and review workflows
Source metadata
Error taxonomy

When to use structured extraction

Use it when software, not a person, consumes the result: a comparison table, a database row, a filter, an alert condition. If the next reader is a human who will skim a passage, cleaned Markdown is cheaper and loses less.

Use, avoid, review manually

Situation	Pattern
Repeating page type with stable fields	Use — schema extraction with validation
One-off question about a single page	Avoid — retrieve and read the passage
Legal, medical, or pricing commitments	Review manually — extract, then require sign-off
Fields the page states only implicitly	Avoid — record null rather than infer

Input design

Extraction quality depends more on what you feed it than on the extraction call. Normalize first: strip navigation and footers, keep headings, tables, and lists, and pass the canonical URL and retrieval timestamp alongside the content. Chunk only when a page exceeds your budget, and chunk on headings so a field and its label do not land in different pieces.

Schema design principles

01Name fields after what the page says, not after your database column.
02Type everything: string, number with unit, enumerated value, ISO date, URL.
03Make every field nullable unless the page type guarantees it.
04Write a one-line description per field — it is the instruction, not decoration.
05Keep enumerations short and add an "other" value with a free-text companion.
06Add source_url and excerpt to every record, and per-field excerpts for anything published.

Illustrative — schema with descriptions and nullabilityjson · 17 lines

{  "type": "object",  "required": ["source_url", "retrieved_at"],  "properties": {    "plan_name":      { "type": ["string", "null"], "description": "Exact tier name as printed on the page" },    "monthly_price":  { "type": ["number", "null"], "description": "Numeric amount only; null if the page says 'contact us'" },    "currency":       { "type": ["string", "null"], "description": "ISO 4217 code if stated or shown as a symbol" },    "billing_period": { "enum": ["monthly", "annual", "other", null] },    "seats_included": { "type": ["integer", "null"] },    "source_url":     { "type": "string" },    "retrieved_at":   { "type": "string", "format": "date-time" },    "excerpts": {      "type": "object",      "description": "Field name to the sentence from the page that supports it"    }  }}

Validation patterns

Checks to run before a record is stored

Check	Catches	Action on failure
Required fields present	Empty or refused extractions	Retry once, then mark incomplete
Type and format	Prices as strings, dates as prose	Coerce if unambiguous, else null with a note
Range plausibility	Decimal and currency slips	Flag for review, never auto-correct
Excerpt contains the value	Values with no support on the page	Drop the value, keep the record
Duplicate detection	Canonical and locale mirrors	Keep canonical, link the duplicates
Cross-field consistency	Annual price on a monthly record	Flag for review

Retry and fallback

Escalation order

011. Re-extract on the same content
Only worth one attempt; a second identical failure is a content problem, not a sampling problem.
022. Re-normalize
Widen what was kept from the page — the value may have been inside a table you stripped.
033. Re-retrieve
Fetch again, and render only if the first fetch shows an interstitial or an empty body.
044. Narrow the ask
Extract the one missing field on its own with a focused instruction.
055. Hand off
Emit an incomplete record with the reason attached. Silence is worse than a gap.

Confidence and review workflows

Model-reported confidence is a weak signal on its own. Prefer observable proxies: was an excerpt returned, did the excerpt contain the value verbatim, did two independent runs agree, did the value change by more than a threshold since the last run. Route records to review when any two proxies disagree.

Source metadata

Every record carries the canonical URL, the retrieval timestamp, the extraction schema version, and the excerpts. Without the schema version you cannot tell whether a change in your data came from the page or from your own prompt. See source metadata patterns.

Error taxonomy

Naming failures so they can be counted

Class	Symptom	Where it is caught
Access	Interstitial, challenge, or login page returned 200	Retrieval verification
Coverage	Right site, wrong page	Selection scoring
Normalization	Value lived in a stripped table	Excerpt check
Ambiguity	Page states two candidate values	Cross-field consistency
Invention	Value with no supporting text	Excerpt-contains-value check
Drift	Layout changed; field silently empty	Fill-rate monitoring over time

Pre-production checklist

Every field has a description and an explicit nullability decision
Excerpts are required for any value that will be shown to a user
Validation runs between extraction and storage
Fill rate per field is monitored so drift is visible
Incomplete records are stored with a reason, not discarded
Schema version is recorded on every record

Practical takeaway

Design the record first, then decide how to fill it.
A field that cannot be null will be invented.
Require an excerpt for every extracted value you intend to publish.
Validation belongs between extraction and storage, not after complaints.