How to Turn a Website Into LLM-Ready Markdown

Raw HTML versus readable semantic content

A modern page is mostly instructions for a browser. Wrapper elements, utility classes, inline styles, analytics payloads, and framework state can account for the overwhelming majority of the bytes. None of it tells a language model anything about the subject of the page.

What does carry meaning is a small set of structures: the heading hierarchy that shows how the content is organised, lists that show enumeration, tables that show relationships, code blocks that must be reproduced exactly, and link targets that show what the page points at. Markdown preserves all of these in a fraction of the space.

What counts as noise

• Primary and secondary navigation, breadcrumbs, and mega-menus
• Cookie and consent notices, region pickers, and app-install prompts
• Footers with sitemap-scale link lists repeated on every page
• Related-content rails, newsletter forms, and share widgets
• Advertising slots and their placeholder text
• Text duplicated from the page title, site name, or section index

The problem with retaining these is not only length. Every page in a site carries the same footer, so a retrieval index built from unnormalized pages contains hundreds of near-identical passages. A query that happens to match footer vocabulary will retrieve an arbitrary selection of them.

What to preserve deliberately

A conceptual before and after

<div class="wrap lg:grid-cols-12">  <nav aria-label="Main"><ul><li><a href="/">Home</a></li>…</ul></nav>  <div class="cookie-bar">We use cookies…</div>  <main>    <h1 class="t-1">Rotating API keys</h1>    <p class="lede">Keys can be rotated without downtime.</p>    <pre><code class="lang-bash">sf keys rotate --id KEY_ID</code></pre>  </main>  <aside class="rail">You might also like…</aside></div>

---title: Rotating API keyscanonical_url: https://example.com/docs/auth/rotationretrieved_at: 2026-09-14T09:41:00Z--- # Rotating API keys Keys can be rotated without downtime. ```bashsf keys rotate --id KEY_ID```

A simple normalization sketch

# Illustrative only — not a runnable API.doc = retrieve(url="https://example.com/docs/auth/rotation", format="markdown") doc = strip_regions(doc, ["nav", "footer", "aside", ".cookie-bar"])doc = protect(doc, ["pre", "code", "table"])   # never drop thesedoc = collapse_repeats(doc)                     # remove site-wide duplicated text record = {    "title": doc.title,    "canonical_url": doc.canonical_url or url,    "content": doc.markdown,    "retrieved_at": now_iso(),} assert len(record["content"]) > 300, "page produced no usable content"

Limitations and edge cases

• Pages whose meaning is visual — charts, diagrams, annotated screenshots — lose information that Markdown cannot carry. Capture the caption and alt text and note the loss.
• Deeply nested or merged-cell tables do not survive flattening cleanly; consider extracting them into structured records separately.
• Single-page applications may render the main content region only after interaction, so normalization rules built for the server response will find nothing.
• Some documentation encodes meaning in tabs or accordions; naive extraction may concatenate mutually exclusive variants into one confusing document.
• Aggressive boilerplate removal can delete short but essential content, such as a one-line deprecation notice in a styled callout.