Tutorial · Research Workflows
How to Build a Vendor Comparison Agent
Evidence-based comparison where 'not established' is a valid answer.
Written by Aaron Grainger
Independent Content Strategist & Product-Marketing Writer · Published Feb 3, 2026
- Primary audience
- Research, intelligence, and editorial teams
- Also useful for
- AI application developers
- Tone
- Educational
- Reading time
- 3 min
- Published
- Feb 3, 2026
Direct answer
Build a vendor comparison agent by writing each criterion as a question that a documentation page could answer, then gathering evidence per vendor per criterion with excerpts. Every matrix cell should be one of: met, partially met, not established, or contradicted — each linked to its source. The design work is in the criteria and the vocabulary mapping, not in the model.
In active development — preview and full outline below
The mistake that makes a comparison matrix confidently wrong
Most automated comparisons fail at one point: they treat a missing page as a negative answer. A vendor that has not documented SSO may not support it, may support it under another name, or may document it behind a login. Collapsing those three into "no" produces a matrix that reads as authoritative and is wrong in a way no reader can see.
Definition: a criterion is a question, not a feature name
"SSO" is a feature name — a reader can argue about whether a cell is a tick. "Does the documentation state that SAML 2.0 single sign-on is available, and on which plans?" is a question a page either answers or does not. The second form makes the cell checkable and makes disagreement productive, because two people can look at the same excerpt.
| Cell | Means | Evidence required | Follow-up |
|---|---|---|---|
| Met | A page states it plainly | Excerpt + URL + retrieval time | None |
| Partially met | Stated with a condition or a tier limit | Excerpt containing the condition | Note the condition in the cell, not the footnote |
| Not established | No retrieved page answers the question | List of pages actually checked | Ask the vendor; this is the cell that generates the call |
| Contradicted | Two pages disagree | Both excerpts, both URLs | Escalate to a human before the matrix is published |
"Not established" is the most useful cell in the matrix because it converts an unknown into a specific question for a sales call. A comparison that never produces it is not being careful — it is guessing and hiding the guess.
Failure modes
| Failure | Cause | Mitigation |
|---|---|---|
| Feature marked absent because it is named differently | No vocabulary mapping | Maintain a synonym list per criterion; keep the vendor's original string in the cell |
| Marketing page treated as documentation | No source-type weighting | Rank docs > changelog > pricing page > marketing; record which tier the evidence came from |
| Matrix favours the vendor with better SEO | Criteria gathered by search rather than from a fixed docs root | Crawl each vendor's documentation root with the same scope rules |
| Matrix is stale at the moment of decision | Evidence gathered weeks earlier | Re-run the criteria set the day the decision is made; show retrieval times |
Full outline
Sections planned for this guide
01Writing criteria as questions
- — Answerable versus unanswerable criteria
- — Evidence standards per criterion
- — Weighting and why it should be declared
02Gathering evidence
- — Documentation before marketing
- — Locating the specific page per criterion
- — Recording excerpts and retrieval times
03The four cell values
- — Met, partially met, not established, contradicted
- — Why 'not established' is the most useful cell
- — Follow-up questions generated from gaps
04Vocabulary mapping
- — Same concept, different names
- — Keeping the original string
- — Where mapping goes wrong
05Presenting the matrix
- — Cell-level citations
- — Showing criteria selection as a choice
- — Refresh before any decision
Practical takeaway
- Criteria written as questions produce checkable cells; criteria written as features do not.
- Absence of a page is not absence of a feature.
- Vendor vocabularies differ; map them explicitly and keep the original wording.
- A comparison is only as objective as its criteria selection, which is subjective.
Related content
Tutorial · 4 min
How to Build a Cited AI Research Agent
A source-first workflow for turning open-web information into accountable AI answers.
Technical guide · 4 min
How to Extract Structured Data From a Website
Use schemas to transform inconsistent pages into dependable records.
Workflow playbook · 6 min
Create a vendor-comparison assistant
Compare vendors on declared criteria using each vendor's own public pages, with disagreement preserved.
Workflow playbook · 6 min
Develop a procurement research assistant
Support a purchase decision with documented, source-linked evidence against written requirements.
Version history
Current: 1.1 · In active development
- 1.0Feb 3, 2026First published.
- 1.1Feb 3, 2026Marked in active development; sections still being expanded.
Was this useful?
Sourceframe is an independent product concept created for research, product-design, and technical-content exploration. It is not an operating company, and nothing here describes a live commercial service. All examples, schemas, and code are illustrative unless a page says otherwise. No client data, customer outcomes, performance results, or partnerships are described anywhere on this site.