Glossary
Shared vocabulary, defined once.
Terms used consistently across every guide, workflow pattern, and research note on this site.
- AI agent
- A model-driven program that plans steps, calls tools, and works toward a goal instead of answering a single prompt.
- Agents differ from chat completions mainly in control flow: they decide what to do next. Web access is one of the most common tools given to an agent, and also one of the most common sources of unreliable behaviour.
- Related: agentic-workflow / web-retrieval / citation
- Web retrieval
- Fetching the content of a known URL and returning it in a form a program or model can use.
- Retrieval covers the request, any rendering required, and the decision about what part of the response is actually content. It is the narrowest of the web-data operations and the easiest to reason about.
- Related: content-normalization / javascript-rendering
- Web crawling
- Following links or a sitemap to retrieve many related pages under a defined scope.
- Crawling adds scope rules, deduplication, and politeness constraints on top of retrieval. Most crawl problems are scope problems: too broad and you drown, too narrow and you miss the pages that mattered.
- Related: sitemap / canonical-url / knowledge-base
- Web scraping
- Pulling specific values out of pages, usually into fields defined in advance.
- Classic scraping binds to markup structure, which breaks when layouts change. Schema-driven extraction binds to meaning instead, which is more tolerant but introduces its own ambiguity.
- Related: structured-extraction / schema
- Browser automation
- Driving a real browser to click, type, scroll, or authenticate in order to reach content.
- Use it when content only exists after interaction. It is slower, more fragile, and more expensive than retrieval, so treat it as an escalation rather than a default.
- Related: javascript-rendering / web-retrieval
- Structured extraction
- Converting page content into records that match a defined schema, with types and optional fields.
- Good extraction records what it could not find rather than guessing. A null with a reason is more useful downstream than a plausible invention.
- Related: schema / source-provenance
- Schema
- A declaration of the fields, types, and requirements an extracted record must satisfy.
- Schemas are a contract between messy input and dependable output. They make validation possible and turn 'the page changed' into a visible failure instead of a silent one.
- Related: structured-extraction / citation
- Markdown
- A lightweight text format that keeps headings, lists, links, and tables without layout markup.
- For most language-model tasks, Markdown is a better carrier than HTML: it preserves the structure that aids comprehension and discards the structure that only serves rendering.
- Related: content-normalization / chunking
- RAG
- Retrieval-augmented generation: supplying a model with retrieved passages so it answers from supplied context.
- RAG quality is dominated by what goes into the index. Retrieval strategy matters, but unusable source content limits the ceiling before any ranking work begins.
- Related: embedding / chunking / knowledge-base
- Embedding
- A numeric vector representing text, used to find passages that are semantically similar to a query.
- Embeddings find related language, not correct answers. They are one retrieval signal among several, and they degrade when chunks mix unrelated topics.
- Related: rag / chunking
- Chunking
- Splitting a document into retrievable pieces sized for indexing and for a model's context window.
- Chunk on document structure where possible — headings, sections, steps — rather than fixed character counts. Carry the page title and section path into each chunk.
- Related: rag / markdown
- Source provenance
- The retrieval record attached to a piece of content: URL, retrieval time, and how it was obtained.
- Provenance is what makes an AI output auditable. Without it, a wrong answer cannot be traced to a wrong source, only to the model.
- Related: citation / structured-extraction
- Citation
- A link from a specific claim to the specific source passage that supports it.
- A citation attached to a paragraph is weaker than one attached to a sentence. Quality is judged by whether a reader can verify the claim quickly, not by the presence of a link.
- Related: source-provenance / ai-agent
- Semantic change detection
- Deciding whether a page difference is meaningful rather than cosmetic.
- Byte diffs alarm on rotating banners and session tokens. Semantic detection compares normalized content or defined fields and ignores the rest.
- Related: content-normalization / web-retrieval
- Sitemap
- A published list of a site's URLs, usually at /sitemap.xml, intended for crawlers.
- Sitemaps are the cheapest possible discovery mechanism for a documentation crawl, but they are frequently stale or partial. Treat them as a starting set, not as truth.
- Related: web-crawling / canonical-url
- Canonical URL
- The address a site declares as the preferred version of a page that is reachable at several URLs.
- Respecting canonicals is the single highest-value deduplication rule in a crawl. Ignoring them is how a 400-page documentation set becomes 4,000 near-identical chunks.
- Related: web-crawling / sitemap
- JavaScript rendering
- Executing a page's scripts so that client-generated content exists in the retrieved document.
- Rendering costs time and compute. Check whether the data arrives in an underlying JSON response or an embedded state blob before paying for a full render.
- Related: browser-automation / web-retrieval
- Content normalization
- Stripping navigation, chrome, and boilerplate so the retained text is the page's actual content.
- Normalization is where most quality is won or lost. Over-aggressive cleaning removes tables and code; under-aggressive cleaning floods the index with cookie banners.
- Related: markdown / rag
- Agentic workflow
- A multi-step process where a model chooses actions, calls tools, and reacts to intermediate results.
- The reliable ones constrain the shape of each step. Free-form autonomy over the open web tends to produce interesting demos and unpredictable products.
- Related: ai-agent / citation
- Knowledge base
- A curated, maintained collection of content prepared for retrieval by people or systems.
- A knowledge base is a maintenance commitment, not a one-time build. Recrawl cadence, removal of dead pages, and review dates matter as much as the initial ingest.
- Related: rag / web-crawling