Sourceframe

Workflow pattern / Extraction

Turn public job pages into labor-market research

Convert public job postings into a structured dataset about declared skills, roles, and locations.

Intermediate / Dataset / Weekly

The problem

Job postings carry useful public signal about what organisations are building, but the pages are inconsistent and quickly removed.

Who it is for

Research and talent teams studying declared demand from public postings.

Teams: Research / Talent / Strategy

Inputs

  • Careers page URLs
  • A job-posting schema
  • Role taxonomy
  • Retention policy

Conceptual process

  1. 01Discover

    Locate careers listings and enumerate posting URLs.

  2. 02Extract

    Apply the posting schema: title, location, employment type, declared skills.

  3. 03Normalize

    Map titles onto a role taxonomy while retaining the original string.

  4. 04Track

    Record first-seen and last-seen so removals are observable.

  5. 05Aggregate

    Summarise by role, location, and skill over time.

Flow diagram

  1. 01Careers pages
  2. 02Enumerate postings
  3. 03Extract schema
  4. 04Normalize roles
  5. 05Track presence
  6. 06Dataset

Example output

Illustrative output
{  "title_raw": "Senior Platform Engineer (Data)",  "role_normalized": "Platform engineering",  "location": { "city": "Toronto", "remote": "hybrid" },  "skills_declared": ["Kubernetes", "Python", "streaming data"],  "first_seen": "2026-08-30",  "last_seen": "2026-09-14",  "source_url": "https://example.com/careers/senior-platform-engineer"}

Data-quality considerations

  • A posting is a declared intention, not a filled role.
  • Keep the raw title alongside the normalized role; taxonomies lose information.
  • Handle personal data carefully — postings can contain named contacts.

Failure modes

  • Reposted listings counted as new demand.
  • Aggregator duplicates inflating counts.
  • Removed postings treated as filled when they may have been cancelled.

Suggested architecture

  • Careers discovery
  • Posting extractor
  • Role normalizer
  • Presence tracker
  • Aggregation layer

What to test first

  1. 01Check duplicate handling across a company that posts to several boards.
  2. 02Verify first-seen and last-seen behave correctly across one removal.

Related reading