Sourceframe

Technical guide · Structured Data

How to Extract Structured Data From a Website

Use schemas to transform inconsistent pages into dependable records.

AG

Written by Aaron Grainger

Independent Content Strategist & Product-Marketing Writer · Published Sep 24, 2024

Primary audience
AI engineers and technical founders
Also useful for
Data and operations teams using public-web information
Tone
Educational
Reading time
4 min
Published
Sep 24, 2024

Direct answer

Extract structured data by defining the schema before touching the pages: fields, types, which are optional, and what units or currencies apply. Run extraction against normalized content, require a supporting excerpt for every value, and allow explicit nulls. Validate the records, quarantine failures rather than dropping them, and track extraction rates per page template so drift becomes visible before it becomes a data-quality incident.

On this page
  1. Why schemas matter
  2. Designing fields
  3. Practical schemas
  4. Extraction sketch
  5. What to do when a field is missing

Why schemas matter

Without a schema, extraction produces text that happens to look like data, and every downstream consumer re-parses it slightly differently. With a schema, the pipeline has a contract: something either satisfies it or fails visibly. That visibility is the entire point — silent wrongness is far more expensive than a rejected record. If you are drafting a field list, the extraction schema starter is a quick way to structure the first pass.

Designing fields

  • Types. Numbers as numbers, dates in one canonical format, enumerations with declared allowed values.
  • Optionality. Mark a field required only if a page without it is genuinely invalid. Over-requiring turns normal pages into failures.
  • Units and currency. Store them as fields next to the value. Never normalize silently across markets.
  • Ambiguity. Where a page can express the same fact several ways, decide which representation wins and record the raw string too.
  • Confidence. A coarse label — high, medium, low — with a stated basis is more usable than a fabricated probability.
  • Provenance. Every record gets source_url and retrieved_at; every field can carry its own excerpt.

Practical schemas

Product listing
{  "sku": "string?",  "name": "string",  "price": { "amount": "number?", "currency": "string?" },  "list_price": { "amount": "number?", "currency": "string?" },  "availability": "in_stock | out_of_stock | preorder | unknown",  "attributes": "object?",  "source_url": "string",  "retrieved_at": "datetime"}
Job posting
{  "title_raw": "string",  "employment_type": "full_time | part_time | contract | unknown",  "location": { "city": "string?", "country": "string?", "remote": "onsite | hybrid | remote | unknown" },  "skills_declared": ["string"],  "posted_label": "string?",  "source_url": "string",  "retrieved_at": "datetime"}
Company profile
{  "name": "string",  "self_description": { "value": "string?", "source_url": "string?" },  "products": ["string"],  "pricing_model": "published | contact_sales | unknown",  "locations_listed": ["string"],  "not_found": ["string"],  "source_url": "string",  "retrieved_at": "datetime"}
News article
{  "headline": "string",  "byline": "string?",  "published_label": "string?",  "summary": "string?",  "canonical_url": "string",  "syndication_of": "string?",  "source_url": "string",  "retrieved_at": "datetime"}
Real-estate listing
{  "listing_id": "string?",  "price": { "amount": "number?", "currency": "string" },  "area": { "value": "number?", "unit": "sqm | sqft" },  "property_type": "string?",  "status": "active | removed | unknown",  "source_url": "string",  "retrieved_at": "datetime"}

Extraction sketch

Illustrative workflow pseudocode
# Illustrative only — not a runnable API.doc = normalize(retrieve(url)) result = extract(    content=doc.markdown,    schema=PRODUCT_SCHEMA,    require_excerpt=True,   # every value must quote its source text    allow_null=True,        # "not present" is a valid answer) record = validate(result, PRODUCT_SCHEMA)if record.errors:    quarantine(record, reason=record.errors)   # never silently dropelse:    store(record | {"source_url": url, "retrieved_at": now_iso()}) track_extraction_rate(template_id=doc.template_id, ok=not record.errors)

Validation checklist

  • Every field has a declared type and an explicit optionality decision
  • Enumerations list their allowed values and include an unknown case
  • Numeric fields are range-checked against plausible bounds
  • Dates parse to one canonical format
  • Currency and unit are present whenever a value depends on them
  • Every value carries the excerpt it came from
  • Records failing validation are quarantined, not dropped
  • Extraction rate is tracked per template
  • A human reviews a sample per template after every schema change

What to do when a field is missing

  1. 01Return null with a reason code — not_present, ambiguous, blocked, or unparseable.
  2. 02Do not infer the value from a sibling page unless the schema explicitly allows it, and then record the inference.
  3. 03Check whether the field is missing for one page or for the whole template; the second case is a rules problem, not a data problem.
  4. 04Decide whether the record is still useful without the field. Often it is, and rejecting it loses real information.
  5. 05Surface missing-field rates in the run report so a slow decline is noticed early.

Questions that keep coming up

Practical takeaway

  • The optional-versus-required decision determines how the pipeline behaves on bad pages.
  • Units and currencies are fields, never assumptions.
  • A null with a reason beats a plausible invention.
  • Review a sample per template, not per record — failures cluster by template.
  • Every record carries source_url and retrieved_at.

Related content

Version history

Current: 1.0 · Published

  1. 1.0Sep 24, 2024First published.

Was this useful?

Sourceframe is an independent product concept created for research, product-design, and technical-content exploration. It is not an operating company, and nothing here describes a live commercial service. All examples, schemas, and code are illustrative unless a page says otherwise. No client data, customer outcomes, performance results, or partnerships are described anywhere on this site.