Sourceframe

Workflow pattern / Extraction

Extract product catalogs into structured records

Turn listing and detail pages into validated records with units, currencies, and source links intact.

Advanced / Dataset / Daily

The problem

Catalog data spread across listing pages, detail pages, and variant selectors is inconsistent and breaks whenever a template shifts.

Who it is for

Data teams building comparison, pricing, or availability datasets from public catalogs.

Teams: Data / Merchandising / Engineering

Inputs

  • Category or listing URLs
  • A product schema with units and currency
  • Pagination and variant rules

Conceptual process

  1. 01Enumerate

    Walk listing pagination to collect detail-page URLs.

  2. 02Retrieve

    Fetch detail pages, rendering only where the data requires it.

  3. 03Extract

    Apply the schema per page, capturing currency and unit as fields.

  4. 04Validate

    Range-check prices, normalize dates, quarantine failures.

  5. 05Track

    Store per-template extraction rates so drift becomes visible.

Flow diagram

  1. 01Category URL
  2. 02Enumerate listings
  3. 03Fetch detail
  4. 04Extract schema
  5. 05Validate
  6. 06Dataset

Example output

Illustrative output
{  "sku": "EX-1042",  "name": "Field notebook, ruled",  "price": { "amount": 18.5, "currency": "USD" },  "availability": "in_stock",  "attributes": { "pages": 128, "binding": "sewn" },  "source_url": "https://example.com/products/ex-1042",  "retrieved_at": "2026-09-14T08:15:00Z",  "missing": ["dimensions"]}

Data-quality considerations

  • Currency and unit are fields, never assumptions.
  • Promotional price and list price are different fields.
  • Sample-review one record per template rather than per product.

Failure modes

  • Variant price captured instead of base price.
  • Out-of-stock items silently dropped rather than recorded as unavailable.
  • Pagination loop ends early because the last page renders differently.

Suggested architecture

  • Listing enumerator
  • Detail fetcher
  • Schema extractor
  • Validator with quarantine
  • Template drift monitor

What to test first

  1. 01Extract fifty products and hand-check ten against the live pages.
  2. 02Confirm an out-of-stock product produces a record, not a gap.

Related reading