I deleted every citation from a research memo then asked an LLM judge to score both the original and the stripped versions. The stripped one scored higher.
Removing all evidence citations across ten memos only moved the judge's score by +0.17. When the memo cites only the evidence that cuts against its own thesis, the score goes up by 0.03. The judge's own run-to-run noise is 0.27. Both defects are invisible to it. A set of typed relation checks caught both at 100%, and told me which object was broken.
Why I ran it
Ontologies for agents are a super hot topic right now, and that's no surprise. However, almost everything written about them is architecture-diagram theoretical. Luckily, ontologies were a major part of a project that I've been involved in recently and the difference they made was palpable. I knew immediately that I wanted to quantify the effect and write up a proper article. This article uses a toy ontology and measures it against the default alternative: asking a strong model to score outputs 1 to 10.
"Ontology" is an overloaded word, so for this article we'll use this definition: a typed inventory of the objects your agent's output claims to contain as well as the relations that must hold between them.
In this experiment we'll use five object types (a memo, market-state claims, theses, trades, risks) and ten relations ("every trade traces to a thesis", "a bearish thesis cites at least one bearish item"). Nothing philosophical, no knowledge graph, no OWL: a JSON Schema and ten Python functions. If you can name your agent's nouns and the ways they can disagree, I guarantee that you can build an ontology.
Everything below is reproducible from the repo: the harness, every memo, every extraction, every judgment, and the audit trail. Fully synthetic data. Total API cost about $35.
The setup
The workload: write an institutional-style research memo from a data packet. The packet has two instruments, a position the account already holds, 60 days of prices, and five news items. One packet leans bearish while the account is long; the other leans bullish while the account is short. The natural memo is "reduce the long" or "cover the short", and there is genuine counter-evidence to weigh in both.
Four models write the memos (Opus 5, GPT-5.6, Kimi K3, Gemini 3.1 Pro, all through OpenRouter, reasoning pinned low). Ten runs each per packet: 80 memos, plus two I wrote by hand as known-good fixtures.
Every memo gets scored twice, from identical information:
- Raw eval: an LLM judge scores it 1 to 10 against a short rubric (analysis, evidence, actionability, risk). Three samples per memo, so the judge's own variance is measured, not assumed.
- Ontology: a model extracts the memo's typed objects into a schema (market_state claims with sources, theses with direction and cited evidence, trades, risks), then ten deterministic checks run over the extraction. Pure Python, no model involved.
A check is a relation between objects, and a failure names its object:
FAIL trade_direction_agrees [tr1] side 'buy' against a bearish thesis
That line is the design argument in miniature. The raw eval outputs a number. The ontology outputs which relation broke, on which object. One of these you can act on.
The news items in the packet carry polarity and scope fields that
make evidence checkable ("is at least one cited item actually negative,
and about this instrument?"). Those fields are the answer key, so the
writer, the extractor, and the judge all get the packet with them
stripped. Nobody being evaluated ever sees them.
Seeding defects
Comparing scores on good memos tells you little: both arms say "fine" and you learn nothing about what happens when an agent breaks. So the real experiment is destructive. I defined nine defect classes, one per relation I care about, before any scoring ran:
- direction_flip: the trade contradicts its own thesis (bearish case, buy order)
- ghost_citation: a thesis cites a news item that doesn't exist
- unsupported_thesis: the thesis cites only evidence that cuts against it
- no_evidence: every citation removed
- wrong_risk: the risk section guards the wrong instrument
- orphan_trade: a second trade with no thesis behind it
- no_risk: the invalidators deleted
- invented_fact: a material claim that appears nowhere in the packet
- ignored_position: all mention of the existing position erased
Each defect gets planted as a prose edit into ten clean base memos (the eight cleanest model-written ones, two per model, plus my two fixtures). Both arms only ever see prose, so the defect has to live in the prose. An LLM applies the edit under a change-exactly-one-thing instruction, every seed ships with its unified diff, and a gate rejects any seed whose diff exceeds a per-defect line budget. The gate fired four times in 90 seeds; I read all four diffs and accepted them (long, citation-dense memos legitimately need more lines to strip).
Ninety seeded memos, each extracted twice and judged three times, next to their clean bases.
The result
("Flagged" = any relation check fired on the seeded memo. Judge
movement is against the paired clean base, noise floor 0.27. Exact
numbers for every class are in the repo's analysis.json.)
Read the top three rows against the bottom five. The judge prices defects that make the prose read wrong. A trade that contradicts its own thesis costs 3.7 points; a suspicious unsourced fact costs 3.3. If a human skimming the memo would frown, the judge frowns.
But the defects that make research untrustworthy rather than awkward are exactly the ones it can't see. A memo that cites nothing reads confident and clean, so it scores fine. Slightly better than fine: the citations were apparently costing style points. A memo whose own citations argue against it reads like a memo with citations. The judge measures "sounds like institutional research". The checks measure "is internally coherent research". These are different properties, and the gap between them is where an agent quietly fails in production.
The judge is a strong current model (Opus 5), not a strawman. And the rubric explicitly asks about evidence grounding. It doesn't matter: a scalar squeezed out of holistic reading keeps rewarding surface.
The clean memos agree
On the 80 unseeded memos, judge score and violation count barely correlate: r = 0.106. And the judge's ranking of the four writers is its own small indictment:
The judge's two favorite writers have seven times the violation rate of the writer it ranks third. Kimi writes the most internally consistent memos in the pool and gets no credit for it, because consistency isn't what the score measures. (The judge is also Opus judging Opus at a flat 8.95 with almost no spread; I didn't measure self-preference properly, so I'll only note the smell.)
Where the ontology arm struggles
Two defect rows are weak, and the reason is worth more than the number.
no_risk (45%) and ignored_position (50%) are diffuse properties.
A well-written memo doesn't keep its risk awareness in the Risk
section: it threads trigger levels, invalidators, and "what makes us
buyers of the rest" through the whole argument. Same for the position:
the good memos are position-aware in their bones. A bounded prose edit
cannot fully remove either property, and the extractor finds the
residue and builds objects from it. The seeding method works for
localized defects and degrades for diffuse ones. That's a limit of my
experiment, not a discovery about ontologies.
Extraction is the arm's load-bearing wall, and it will lie to you politely if you let it. My first extraction prompt said "extract what the memo says". The extractor, holding the packet, quietly repaired broken memos: it back-filled citation ids it recognized from the news text and marked position-awareness the memo didn't have. Two defect classes came back at 0% detection because the extraction layer had fixed the defects on the way through. The fix was prompt-level brute force ("if the memo describes an event but does not cite its id, do NOT add the id") plus measuring extraction stability across two samples per memo (stable check-sets on 149 of 172). If you run checks over an extraction, audit the extraction. Transcription is not repair.
What I'd actually take to production
The toy maps onto any agent whose output is "recommend an action from evidence": support triage, code-review bots, medical intake, deal memos. The objects rename; the shape survives. Claims with sources, judgments with direction and evidence, actions that trace to judgments, guards that watch the judgments. The checks are relations between them, and every one is a boring pure function.
What the two arms are for, after running both: the judge is a smoke
detector for prose quality, and it's decent at that. The relation
checks are the instrument for the failures that matter, because they
catch the invisible class, they localize ([tr1], not "could be more
thorough"), and they regression-test for free: a check that fires
today fires tomorrow, while a judge score drifts with the judge.
Build order if you're starting from zero: define the objects your agent's output claims to contain, write the five relations that would embarrass you most if they broke, and only then argue about rubrics. The extraction pass costs one model call and the checks cost nothing. My whole experiment, 80 generated memos, 90 seeded ones, 344 extractions, 516 judgments, came to about $35.
Harness, data, and every audit diff: antithetical-labs/llm-research-evals.