Removing all evidence citations across ten memos only moved the judge's score by +0.17. When the memo cites only the evidence that cuts against its own thesis, the score goes up by 0.03. The judge's own run-to-run noise is 0.27. Both defects are invisible to it. A set of typed relation checks caught both at 100%, and told me which object was broken.

Why I ran it

Ontologies for agents are a super hot topic right now, and that's no surprise. However, almost everything written about them is architecture-diagram theoretical. Luckily, ontologies were a major part of a project that I've been involved in recently and the difference they made was palpable. I knew immediately that I wanted to quantify the effect and write up a proper article. This article uses a toy ontology and measures it against the default alternative: asking a strong model to score outputs 1 to 10.

"Ontology" is an overloaded word, so for this article we'll use this definition: a typed inventory of the objects your agent's output claims to contain as well as the relations that must hold between them.

In this experiment we'll use five object types (a memo, market-state claims, theses, trades, risks) and ten relations ("every trade traces to a thesis", "a bearish thesis cites at least one bearish item"). Nothing philosophical, no knowledge graph, no OWL: a JSON Schema and ten Python functions. If you can name your agent's nouns and the ways they can disagree, I guarantee that you can build an ontology.

Everything below is reproducible from the repo: the harness, every memo, every extraction, every judgment, and the audit trail. Fully synthetic data. Total API cost about $35.

The setup

The workload: write an institutional-style research memo from a data packet. The packet has two instruments, a position the account already holds, 60 days of prices, and five news items. One packet leans bearish while the account is long; the other leans bullish while the account is short. The natural memo is "reduce the long" or "cover the short", and there is genuine counter-evidence to weigh in both.

Four models write the memos (Opus 5, GPT-5.6, Kimi K3, Gemini 3.1 Pro, all through OpenRouter, reasoning pinned low). Ten runs each per packet: 80 memos, plus two I wrote by hand as known-good fixtures.

Every memo gets scored twice, from identical information:

• Raw eval: an LLM judge scores it 1 to 10 against a short rubric (analysis, evidence, actionability, risk). Three samples per memo, so the judge's own variance is measured, not assumed.
• Ontology: a model extracts the memo's typed objects into a schema (market_state claims with sources, theses with direction and cited evidence, trades, risks), then ten deterministic checks run over the extraction. Pure Python, no model involved.

A check is a relation between objects, and a failure names its object:

FAIL trade_direction_agrees [tr1] side 'buy' against a bearish thesis

That line is the design argument in miniature. The raw eval outputs a number. The ontology outputs which relation broke, on which object. One of these you can act on.

The news items in the packet carry polarity and scope fields that make evidence checkable ("is at least one cited item actually negative, and about this instrument?"). Those fields are the answer key, so the writer, the extractor, and the judge all get the packet with them stripped. Nobody being evaluated ever sees them.

Seeding defects

Comparing scores on good memos tells you little: both arms say "fine" and you learn nothing about what happens when an agent breaks. So the real experiment is destructive. I defined nine defect classes, one per relation I care about, before any scoring ran:

1. direction_flip: the trade contradicts its own thesis (bearish case, buy order)
2. ghost_citation: a thesis cites a news item that doesn't exist
3. unsupported_thesis: the thesis cites only evidence that cuts against it
4. no_evidence: every citation removed
5. wrong_risk: the risk section guards the wrong instrument
6. orphan_trade: a second trade with no thesis behind it
7. no_risk: the invalidators deleted
8. invented_fact: a material claim that appears nowhere in the packet
9. ignored_position: all mention of the existing position erased

Each defect gets planted as a prose edit into ten clean base memos (the eight cleanest model-written ones, two per model, plus my two fixtures). Both arms only ever see prose, so the defect has to live in the prose. An LLM applies the edit under a change-exactly-one-thing instruction, every seed ships with its unified diff, and a gate rejects any seed whose diff exceeds a per-defect line budget. The gate fired four times in 90 seeds; I read all four diffs and accepted them (long, citation-dense memos legitimately need more lines to strip).

Ninety seeded memos, each extracted twice and judged three times, next to their clean bases.

The result

("Flagged" = any relation check fired on the seeded memo. Judge movement is against the paired clean base, noise floor 0.27. Exact numbers for every class are in the repo's analysis.json.)

Read the top three rows against the bottom five. The judge prices defects that make the prose read wrong. A trade that contradicts its own thesis costs 3.7 points; a suspicious unsourced fact costs 3.3. If a human skimming the memo would frown, the judge frowns.

But the defects that make research untrustworthy rather than awkward are exactly the ones it can't see. A memo that cites nothing reads confident and clean, so it scores fine. Slightly better than fine: the citations were apparently costing style points. A memo whose own citations argue against it reads like a memo with citations. The judge measures "sounds like institutional research". The checks measure "is internally coherent research". These are different properties, and the gap between them is where an agent quietly fails in production.

The judge is a strong current model (Opus 5), not a strawman. And the rubric explicitly asks about evidence grounding. It doesn't matter: a scalar squeezed out of holistic reading keeps rewarding surface.

The clean memos agree

On the 80 unseeded memos, judge score and violation count barely correlate: r = 0.106. And the judge's ranking of the four writers is its own small indictment:

The judge's two favorite writers have seven times the violation rate of the writer it ranks third. Kimi writes the most internally consistent memos in the pool and gets no credit for it, because consistency isn't what the score measures. (The judge is also Opus judging Opus at a flat 8.95 with almost no spread; I didn't measure self-preference properly, so I'll only note the smell.)

Where the ontology arm struggles

Two defect rows are weak, and the reason is worth more than the number.

no_risk (45%) and ignored_position (50%) are diffuse properties. A well-written memo doesn't keep its risk awareness in the Risk section: it threads trigger levels, invalidators, and "what makes us buyers of the rest" through the whole argument. Same for the position: the good memos are position-aware in their bones. A bounded prose edit cannot fully remove either property, and the extractor finds the residue and builds objects from it. The seeding method works for localized defects and degrades for diffuse ones. That's a limit of my experiment, not a discovery about ontologies.

Extraction is the arm's load-bearing wall, and it will lie to you politely if you let it. My first extraction prompt said "extract what the memo says". The extractor, holding the packet, quietly repaired broken memos: it back-filled citation ids it recognized from the news text and marked position-awareness the memo didn't have. Two defect classes came back at 0% detection because the extraction layer had fixed the defects on the way through. The fix was prompt-level brute force ("if the memo describes an event but does not cite its id, do NOT add the id") plus measuring extraction stability across two samples per memo (stable check-sets on 149 of 172). If you run checks over an extraction, audit the extraction. Transcription is not repair.

What I'd actually take to production

The toy maps onto any agent whose output is "recommend an action from evidence": support triage, code-review bots, medical intake, deal memos. The objects rename; the shape survives. Claims with sources, judgments with direction and evidence, actions that trace to judgments, guards that watch the judgments. The checks are relations between them, and every one is a boring pure function.

What the two arms are for, after running both: the judge is a smoke detector for prose quality, and it's decent at that. The relation checks are the instrument for the failures that matter, because they catch the invisible class, they localize ([tr1], not "could be more thorough"), and they regression-test for free: a check that fires today fires tomorrow, while a judge score drifts with the judge.

Build order if you're starting from zero: define the objects your agent's output claims to contain, write the five relations that would embarrass you most if they broke, and only then argue about rubrics. The extraction pass costs one model call and the checks cost nothing. My whole experiment, 80 generated memos, 90 seeded ones, 344 extractions, 516 judgments, came to about $35.

Harness, data, and every audit diff: antithetical-labs/llm-research-evals.

So: what does move it. Two interventions, same briefs, same four models, five runs each.

• forbid: name the four commonest patterns and ban them. "Do not use gradients of any kind. Do not use a dark background with a single bright accent colour. Do not use frosted-glass or blur effects. Do not use fully-rounded pill-shaped buttons."
• direct: one sentence pointing somewhere. "Take the visual direction from acid design, futuristic, digital, high-energy."

I said at the end of part 1 that one of these produces pages with no personality whatsoever, and that it's the one I expected to work. It's the ban, and it does exactly what I asked it to.

The ban is obeyed

The four named patterns land at 0%, 0%, 0% and 3%. One page in 39 slipped a gradient through and that is the entire violation set. "Models are bad at negation" is not true here, at least for concrete visual features you can name.

Nothing I didn't name lands anywhere near zero either, with one honest exception: emoji fell to 5%, two pages in thirty-nine, without being mentioned. Everything else unnamed sits between 15% and 74%.

The patterns come in bundles

Look at glow shadow, though. It drops 42 points and I never mentioned it.

That makes sense once you notice a glow needs a dark background to be visible, and dark backgrounds were banned. But it turns out to be the general case, not a coincidence:

These features are not independent. A pill button and an eyebrow label have nothing to do with each other visually, one is a button shape and the other is a line of small caps, and they land on the same page 67.5% of the time. Their phi correlation is 0.82, and all eight of the strongest pairs sit above 0.4.

The obvious objection is that this is the product category showing through: dev tools are dark and glowy, back-office tools are rounded and light, so of course things co-occur. Splitting by brief doesn't collapse it. The grey dots are each brief's own value, 0.88 and 0.79 for pill and eyebrow, and the weakest of the sixteen is 0.36, nowhere near zero.

So an attractor isn't a feature, it's a package. Cut one strand and you pull the others, which is why a ban on four things reaches a fifth.

What you land on instead

Ban the package and the model doesn't run out of ideas. It picks up a different package and holds it just as tightly.

The two patterns that came through untouched are pure page furniture, an eyebrow label over a heading and numbered section markers, and both went up. Meanwhile every forbidden page is white, in both briefs, with a within-brief spread in background lightness of 0.009 and 0.012 (standard deviation). Tighter than any other condition in the experiment, and only the high-effort rerun of the same prohibition beats it.

Top row is the ban. They're competent and they're interchangeable. I can't tell you which model made which, and neither could you.

They're also the emptiest pages in the experiment: median 24 coloured elements against 35 untouched.

The category doesn't vanish here either, it relocates. The prohibition names background colour, so matching backgrounds prove nothing on their own. But under the identical ban, on identical white pages, the dev tool still carries twice the colour and two thirds the page length:

Everything got quieter. Nothing got resolved.

Is that just the small thinking budget?

Reasoning is pinned low across this experiment, which leaves an obvious reading: a constrained model on a small budget takes the cheapest compliant path, and the white page is laziness rather than a destination.

So I ran the identical prohibition again with effort raised. Same prompt, byte for byte, one knob. It came out at 33 times the reasoning, a median of 2,060 tokens a page against 63, across 24 pages.

The palette doesn't move. Every page lands between 0.952 and 0.994, dark backgrounds stay at 0%, and the banned patterns stay gone. What the thinking buys is bulk: 84% more elements, 54% more colour, a fifth more page. The model builds a more thorough version of the same white document.

It's the attractor, not the budget.

What one sentence of direction buys

Now the other half.

Before writing any markup, the directed models were asked to say what the aesthetic actually is. Forty documents, median 717 words, written before a single tag. I expected four different guesses at a vague phrase.

Every single document reaches for rave flyers. Thirty-nine of forty reach for monospace, thirty-eight name Y2K, thirty-six name chrome and thirty-six name magenta, and thirty-three place it in the nineties. Seven cite The Designers Republic by name. One of Claude's runs goes further and lists David Rudnick, Bráulio Amado, Studio Moross and the Pangram Pangram foundry.

They are not guessing. They are recalling the same movement:

Acid graphics are the visual residue of three lineages colliding. First, rave and jungle flyer culture of the early-to-mid nineties: photocopied, over-saturated, illegible-on-purpose, printed on cheap stock in fluorescent spot inks. Second, chrome-era 3D […]. Third, the anti-design / post-Swiss reaction of the 2010s […]

That's Claude. Here's GPT, unprompted by any of it:

Start with a near-black base—charcoal, ink, or a subtly tinted ultraviolet black—rather than pure black. Use a small set of synthetic accents at full intensity: toxic lime, electric violet, hot magenta, signal cyan, and occasional warning orange.

Which explains why the direction works at all. I wasn't asking for invention. I was naming a coordinate the training data holds densely, and four models walked to it from four different starting points.

And you can watch them arrive. Untouched, the accent hue is all over the place: across their ten runs each model reaches for three or four different hue families, so there is no single "favourite" to report. All forty untouched pages split green 15, purple 11, orange 11, magenta 3.

Directed, 32 of those 40 pages land in one family. Claude 8 out of 10, kimi and gemini 9, gpt 6. All four go 100% dark. Pages carrying a genuinely saturated colour, above 0.25 max chroma, go from 8% untouched to 70%, and coloured elements from 35 to 73.

The vendor fingerprint I spent half of part 1 documenting turns out to be about a finger's pressure of instruction deep. Not all of it: Claude still put an eyebrow label on 100% of its directed pages, exactly as it does untouched, while its pill buttons dropped from 100% to 10%. Gemini still refused eyebrows entirely. Direction overrode the palette and left the component vocabulary alone, which is exactly the split part 1 predicted.

One thing the numbers hide: that family is magenta, and the pages look green. My extractor takes the highest-chroma element as the accent, and on an acid palette the magenta beats the acid green that dominates the page.

Does the grounding step earn its turn?

Everything above is the grounded pipeline. The obvious question is how much of it the grounding turn is doing, so: direct1 gets the identical sentence and goes straight to HTML, no writing-it-down step.

Nearly three times tighter run to run, and a third more on the page.

The single clearest thing the grounding turn does is visible per model. Given just the sentence, GPT went dark on one of its ten pages; the other nine came back light. Given the turn to write down what acid design is first, it went dark on all ten. It isn't that GPT couldn't do acid design. It's that "take the visual direction from acid design" alone didn't reliably get it there, and four hundred words of its own prose did.

Peak saturation went slightly down with grounding, 0.276 to 0.256. Thinking first made the pages more considered, not more extreme.

The model is going to converge either way

Every condition here converges. Nothing said, banned, directed: each lands the four models in more or less the same place as each other, and the tightest convergence of the lot belongs to the ban.

So convergence was never the problem. The question is only ever who picked the destination.

Banning strips a layer. You peel off the package the model reaches for first and it settles onto the next one down, which nobody chose and which happens to be a white page with an eyebrow label. Direction names the destination, and because the destination is something the training data holds richly, one sentence is enough to move four models off four different defaults.

Which is why "make it less generic" has never worked for me, and I now know what it costs rather than just that it doesn't work. It's subtraction with no target. The slop leaves, exactly as instructed, and something equally fixed arrives to take its place.

What this doesn't show

• One direction, tested once. Every directed page here aims at acid design, and the grounding documents are the reason it worked: the models already hold a dense, shared account of it. A direction the training data holds thinly would be the real test and I haven't run it.
• "High effort" is not one treatment. At the same setting kimi spent 5,690 reasoning tokens a page and gpt spent 336. The comparison holds across all four, but the knob plainly means different things to different vendors.
• Reasoning is pinned low everywhere except that one A/B, so the direction results have not been checked against a bigger thinking budget.
• The format bans Tailwind, same as part 1, same caveat.
• 39 forbidden pages, not 40. One cell never completed, twice, and the harness raises rather than writing a partial row, so the failure is described in the code and not in the published runs.jsonl.

Harness, all 263 pages, the grounding documents and every measurement: antithetical-labs/llm-frontend-evals.

Spoiler for the whole post: the pages are good. Genuinely better than what I'd put together by hand. They're just all the same page. Which shouldn't surprise anyone, because a model reaching for the most likely next token is going to reach for the most likely design, and "be distinctive" is not a direction you can point that at. Distinctive isn't a thing you can be. It's a thing you can be towards.

The setup

Four models, all through OpenRouter on one key, mostly so I wouldn't have to babysit four billing dashboards:

• Claude Opus 5
• GPT-5.6-sol
• Kimi K3
• Gemini 3.1 Pro

Every model gets the same two briefs, and these stay fixed for the whole series:

• Streamforge, a dev tool. Live-streaming platform where developers broadcast themselves building software with AI assistants. Viewers watch the prompts, the model's output and the code evolve in real time, and can fork any stream's repo.
• Backlot, a back-office tool. Brand-deal contracts, invoices, sponsor deadlines and per-platform payouts for full-time content creators who are not finance people.

Two of them, rather than one, to separate what a model always does from what the product pulls out of it. Short version: the palette flips, the components don't.

Output format is pinned everywhere: one self-contained HTML document, all CSS in a single <style> block, no frameworks, no build step, and a fixed set of sections. Six prompting conditions, five runs each. 240 cells, 239 completed, $44.39.

Harness, raw pages and every measurement: antithetical-labs/llm-frontend-evals.

This post is the first two conditions: saying nothing, and saying "avoid generic AI aesthetics."

What does a model do when you say nothing?

The briefs don't mention colour, type or layout. Not once. Here's the background lightness of all 40 untouched pages:

So it's not a dark mode preference. It's genre retrieval: dev tool means dark, back-office tool for non-technical people means light and editorial. The model looks up what your category is supposed to look like and ships that. Confident as hell about a product it met ninety seconds ago.

The copy converges too. Nineteen of twenty dev-tool headlines open with "Watch", and three vendors independently wrote a sentence within one word of "Watch software get built, prompt by prompt." That one's on me though: my brief says "Viewers watch the prompts", so the word was already sitting there. The back-office headlines, no such verb in their brief, share zero words across models. Still funny how far a single word drags the copy behind it.

Do the four models actually differ?

Genre sets the palette. Underneath that they're pretty different. Coral is the model's own rate, grey is the four-model average behind it:

Gradients are universal, ~90% everywhere. Everything else splits:

• Gemini just doesn't do eyebrows. That little uppercase kicker above a heading: 7 of its 60 pages. The other three run 55 to 58 out of 60.
• Gemini won't make a pill button either. Zero out of ten. Claude and Kimi are at 100%. Somewhere in that post-training there is an opinion about border-radius.
• Claude cannot leave a surface alone. Gradient, frosted glass and pill on 100% of its untouched pages.
• Kimi is the only one that reaches for a serif headline, and only sometimes (3 of 10).
• Page weight varies 3x. Gemini's median page is 88 DOM elements. GPT's is 261.

Which is the actually useful bit here. If you switch models and the output feels different in a way you can't name, this is it. The palette follows your product no matter who you ask. The component vocabulary is your vendor's.

So there isn't one most likely page. There are four, one per vendor. Each of them still only has the one, though.

Is the slop checklist still right?

There are real lists for this, not just vibes. pbakaus/impeccable ships 59 detector rules and opens by naming the tells directly: "Inter for everything, purple-to-blue gradients, cards nested in cards, gray text on colored backgrounds, the rounded-square icon tile above every heading." MindStudio's design-system post adds blue or indigo accents, rounded-full pills, 50px padding everywhere, drop shadows, glassmorphism and gradient hero sections.

Six of those I can measure. Scorecard:

On texture they're dead right. Gradient 90%, pill 72%, glassmorphism 62%, shadow or glow 57%. No notes.

On colour they're stale. "Blue or indigo as the primary accent" lands on 28% of pages. Green and acid are the biggest group at 38%, red and orange next at 32%. Purple-to-blue was probably correct whenever those lists were written. Acid green has moved in since.

Which is the structural problem with ban lists. The mode moves. Whatever was most likely last year got written down, and by the time it's a rule the models have drifted somewhere else. You're always describing the previous convergence.

The font one I can't score fairly, and I want to be upfront about it: my output format required a system font stack, so I discouraged Inter myself. 30% of pages put it in the stack anyway.

Then there's the half nobody names. Eyebrow labels 70%, dingbats 60%, numbered section markers 52%. Not on either list.

The dingbat row is my favourite. 60% of untouched pages have a ✓ or a ✦ in the markup, mostly ticking off feature bullets. Real emoji are way rarer at 22%. I nearly shipped these as one number, which would have published the checkmark's frequency under the rocket's name.

Here's one page carrying nearly all of it:

Three eyebrows ("BUILT FOR PEOPLE WHO SHIP", "HOW IT WORKS", "OPEN BETA"), pill buttons, a glass card over a gradient, a glow on the accent, three numbered steps, checkmarks down the feature list. Accent is orange, not indigo. Run it past a detector looking for purple-to-blue and it comes back clean.

As my own sidenote, I originally recorded zero rounded corners across all 239 pages. Great line. Completely wrong: I was taking the median corner radius over every large element, and only ~10% of large elements are rounded, so the median never sees them. Rounding is sparse, not absent.

That one had a second life, which is the more useful lesson. The classifier in the next section tests for "zero border-radius", and it was testing it with that same median. So the check passed on every page in the run, and the bucket that was supposed to mean broadsheet quietly became a catch-all holding 65 pages, some of them 45% rounded. A dead measure doesn't just give you one wrong number. It gives you every number computed downstream of it, and those look fine.

Does "make it distinctive" do anything?

Obvious next move: just tell it not to. The folk remedy, in the wording people actually use, is "avoid generic AI-generated aesthetics, make it distinctive." I appended exactly that and ran the 40 pages again.

The polish comes off hard. Pill buttons drop 57 points, frosted glass 45.

The right panel needs a word of setup. Anthropic ships a frontend-design skill with Claude that names three looks AI design falls into: cream-and-serif, near-black-with-one-bright-accent, and broadsheet. I classified every page against those three.

The nudge trades between them and doesn't get you out. Dark-and-neon halves, 25 pages down to 13, and broadsheet picks up 10 of the 12 it loses. The number sitting outside all three barely moves: 14 untouched, 13 nudged.

That list is missing one, though. Every near-white back-office page is hairline-ruled and dense but has rounded cards, which is the opposite of broadsheet, so it matches none of the three. That's just the ordinary light SaaS look, and it's most of what the second brief produces.

Two things go up. Numbered markers +30 points, eyebrows +5, median DOM elements 124 to 173. Told to be distinctive, every model added more stuff. "Be distinctive" parses as "add something."

Same model, same brief, one sentence of difference:

Still dark. Still one bright green accent. Buttons went square, glass card gone, type went mono, and there's a strikethrough gimmick in the headline. It looks different. It doesn't look like a different decision, it looks like the same decision with the gloss sanded off.

Which is about what I'd expect. Telling a model what not to do is a constraint, and a constraint narrows the distribution without moving it. You get the same design with fewer features on it. Removal is not direction.

And this is the whole thing, I think. None of these models is bad at design. They build a clean, competent, well-spaced page every single time, which is more than I manage. What they can't do is be distinctive on request, because distinctive means "away from the mode" and nothing in the prompt says which way. Ask for the most likely page and you get it. Ask for a less likely page and you still get the most likely one, minus the pill buttons.

What this doesn't show

• Reasoning is pinned low on every model. At default settings one model burned 16k tokens thinking before writing a single tag and another took under a minute. Uncontrolled variable, not a feature.
• Plain HTML and CSS only, no Tailwind, no React. Utility classes push colour and spacing into class names and a framework's default scale, which is the exact thing I'm measuring. Most real frontend work happens in Tailwind, so this is a deviation from practice and I'd rather say so.
• Two briefs separate genre from model. They don't characterise a genre. Everything above about dev tools going dark rests on exactly one dev tool.
• One cell never finished, dying identically twice at 16,380 bytes with three reported completion tokens. Left it as a gap.

Next up, the obvious follow-up: if vague negation does nothing, does a specific one work? I banned the four patterns by name, and separately gave one sentence of actual direction to aim at. One of those produces pages with no personality whatsoever. It's the one I expected to work.