Strategic guide · Technical SEO and Content Intelligence

How to Turn a Sitemap Into a Searchable Content Dataset

From a URL list to a queryable table of pages, topics, and structure.

Independent Content Strategist & Product-Marketing Writer · Published Dec 9, 2025

Primary audience: Growth and content leaders at developer-tool companies
Also useful for: Developer advocates and technical writers
Tone: Authoritative
Reading time: 2 min
Published: Dec 9, 2025

Direct answer

Turn a sitemap into a dataset by parsing the URL list, retrieving and normalizing each page, and extracting a consistent row per page: title, heading structure, word count, internal links, canonical URL, and topic. The result is a table you can query for coverage gaps, duplicate intent, orphan pages, and thin content. The main caveat is that sitemaps are frequently incomplete.

In active development — preview and full outline below

On this page

The cheapest useful content-intelligence exercise
Definition: the question a page answers
Limits of the sitemap as a source

The cheapest useful content-intelligence exercise

A few hundred rows describing your own site, produced in an afternoon, reliably surfaces two things teams did not know: pages that answer the same question twice, and pages nothing links to. Neither requires a model — they are GROUP BY queries over a table you already have the means to build.

Definition: the question a page answers

The most valuable column is also the only subjective one. Record, in one sentence, the question a reader arrives with. Two pages whose sentences paraphrase each other are duplicate intent regardless of how different their keywords look — and duplicate intent, not thin content, is what usually splits a site's coverage.

Illustrative rowjson · 10 lines

{  "url": "https://example.com/guides/rag-basics",  "title": "RAG basics",  "h1": "RAG basics",  "h2_count": 6,  "words": 1840,  "internal_links_in": 3,  "canonical_url": "https://example.com/guides/rag-basics",  "question_answered": "What is RAG and when should it be used?"}

Queries worth running the day the table exists

Question	Query shape	What it finds
Which pages are orphans?	`internal_links_in = 0`	Pages readers and crawlers cannot reach by navigation
Where is intent duplicated?	Group by `question_answered`, count > 1	Consolidation candidates
Which pages are thin?	`words < 400 AND h2_count < 2`	Stubs that dilute a topic cluster
Where does canonical disagree?	`canonical_url != url`	Duplicate or parameterised URLs in the index
Which hubs carry the site?	Order by `internal_links_out` desc	Pages worth keeping accurate

Limits of the sitemap as a source

Assumption	Reality	Correction
The sitemap lists every page	Generated sitemaps commonly omit paginated, filtered, and legacy URLs	Union the sitemap with a scoped crawl and compare the two sets
Listed URLs resolve	Entries persist after deletion and redirect	Record final URL and status; drop non-200s into a separate table
`lastmod` means edited	Many systems stamp it on every deploy	Ignore it; use content hashes if you need change detection
One sitemap file	Index sitemaps nest several	Parse recursively before assuming a small site

Full outline

Sections planned for this guide

01Parsing the sitemap
- — Index sitemaps and nested files
- — Stale and redirecting entries
- — Filling gaps with a scoped crawl
02Building the row
- — Columns worth having
- — Heading structure as a quality signal
- — Word count caveats
03Link graph
- — Internal links in and out
- — Finding orphans
- — Hub pages and their value
04Topic clustering
- — Clustering by question answered
- — Detecting duplicate intent
- — Identifying real gaps
05Using the dataset
- — Editorial planning
- — Consolidation candidates
- — Refresh cadence

Practical takeaway

A sitemap is a starting set, not an inventory.
One row per page with consistent columns makes gap analysis trivial.
Internal link counts reveal orphan pages faster than any crawl report.
Cluster by the question a page answers, not by keyword.