The creative engine: how to make judgment reliable

Written 21 Aug 2026 after Alex reframed the goal: *"the real task and goal is for the backend to work well as individual apps even. meaning i want each function to outperform most of whats on the market due to super good logic, agents, and orchestrators… the creatives and strategy needs to be so on point people would pay tens of thousands for it."*

That is a specific bar, and it is measurable. Human creative strategy sells for $5,000–25,000 a month. So the competitor is not AdCreative.ai, it is a senior strategist — and the question is what a system can do that a strategist cannot, rather than how to imitate one.

The number that should decide where the effort goes

Creative strategy — concept, angle and brief quality — accounts for 50–60% of creative performance. Execution quality accounts for 30–40%. Hook quality 10–20%.

Atlas is a production company, so the instinct is that execution is the lever. It is a third of the lever. This is also a second, independent argument for Higgsfield first: we should not be building a renderer, because rendering is not where performance lives.

And the corollary, stated by the market itself: *an agency that leads with aesthetics and cannot produce creative-level CPA data is a creative agency with performance branding.* We run the ads, so we can produce that data. Almost nobody selling creative software can.

The real problem: `taste.md` requires a rejection function, and naive judges are unreliable

rules/taste.md is right that taste is a rejection function and that slop is averaging. But "generate wide, kill hard" only works if the killing is *trustworthy*. If the judge is biased, generating wider amplifies the bias rather than escaping it — you get more confident slop.

The 2026 literature on LLM-as-judge is unambiguous about the failure modes:

Frontier models fail more than half of bias tests. Position bias, verbosity bias, and a systematic preference for style over factual accuracy.
Position bias survives rubrics. Rubric-based judging behaves like a multiple-choice question, and models prefer options at particular positions in the list. Permuting option order measurably corrects it.
Pairwise comparison reaches higher human agreement than absolute scoring, because the judge compares rather than holding a calibrated standard across runs, models and months.
Panels conform. In multi-agent debate, weak models correct only 3.6% of stance biases — agents abandon correct judgments to agree with the majority. A "panel of critics discussing" is therefore worse than it looks: it converges, and convergence reads as confidence.
Debate helps under a precise condition: when the critic's verification odds, weighted by fixability, exceed the generator's baseline accuracy odds. That condition predicted all nine task types in the study with zero false positives across nineteen published comparisons.
Three critique-and-revision rounds is usually enough. Past that, diminishing returns.

Those six findings dictate the design. Most of what people build — one blended critic scoring 1–10, or a round-table of agents talking to each other — is precisely what the evidence says fails.

The architecture

Six layers, each justified by one of the findings above.

0. Deterministic gates — free, absolute, first

Already built: diversity.ts, payoff.ts, research/usefulness.ts.

The literature's own split supports this: *rubric scoring is the right primitive for absolute gates; pairwise is the right primitive for relative quality.* So checkable rules stay in code, never in a model. They cost nothing, never drift, and stop a model being asked to notice what a regex knows.

1. Generate diverse by construction, not by rejection

Killing duplicates is second best. The better move is to make duplication impossible: generate one concept per persuasion mechanism, forced, so a set of eight is eight different tricks before any judgment happens. Rejection then handles what construction cannot — a mechanism executed badly.

This inverts the usual order and is cheaper: it spends generation budget on coverage rather than on twelve variations of the safest idea.

2. Criterion-separated judges, run blind and independently

Never one judge scoring "quality". taste.md already argues this from first principles — a single pass optimises a blend and lands on the median — and the bias research now supports it empirically, since blended judges are where style-over-substance preference shows up.

So: one judge per criterion (proof-backing, awareness fit, mechanism strength, specificity, voice), each seeing only its own criterion, none seeing another's verdict. That last clause is the important one. It is the direct answer to the 3.6% conformity finding: judges that cannot see each other cannot converge on a shared error. Aggregation is mechanical, not conversational.

3. Pairwise tournament for selection

Not "score each concept, take the top four". Pairwise, because pairwise agrees with humans more often, and because absolute scores drift between runs and models in a way that makes month-to-month comparison meaningless.

Every comparison is run both ways and disagreement between the two orderings is recorded as an abstention rather than resolved. A judge that prefers whichever concept it saw first is telling us it has no opinion, and treating that as a preference is how position bias becomes a shipped decision.

4. Bounded critique — and only where it pays

Cap revision at three rounds.

More importantly, apply the debate-value condition rather than adding critics reflexively: a critic earns its place only where verification is genuinely easier than generation. That is true of proof-backing (checking a claim against a fact is easier than writing a claim) and false of voice (judging whether copy sounds like the client is about as hard as writing it). So proof gets a critic and voice gets a human — which is a decision the condition makes for us, rather than a preference.

5. The terminal judge: what actually happened

Every layer above is a proxy. The only real judge is performance, and this is the layer no creative SaaS can build for a specific client, because they do not run the ads.

Requirements:

Lineage on every creative. Which angle, which mechanism, which proof, which format, which awareness stage. You cannot attribute a result to a concept you did not tag, and the point is to attribute performance to the decision, not to the asset.
Hook and hold rate per concept, since those two together predict CPA better than either alone.
Platform attribution weekly, incrementality quarterly. The gold-standard pattern is platform numbers for day-to-day optimisation, validated periodically by a real holdout — 90/10 split — so the weekly numbers do not quietly drift away from reality.
Back-propagation into the judges. Which criteria actually predicted performance? Which mechanisms win for this industry? A judge whose verdicts do not correlate with outcomes gets demoted. This is the difference between a rubric someone wrote and a rubric that has been right.

That loop is the whole argument for the price. A strategist has taste and memory. This has taste, memory, and a record of every judgment it has ever made against what happened next.

What this means for the roadmap

The engine is worth more than the portal, and the portal is the interface to it. In priority order:

The judge harness — pairwise, blind, criterion-separated, position-permuted, mechanical aggregation. *(Built: src/lib/creative/tournament.ts)*
Generation by mechanism coverage, so diversity is structural.
Creative lineage, without which layer 5 is impossible. Cheap now, expensive later — every creative shipped untagged is a permanently unattributable data point.
Hook and hold ingestion from Meta, per concept.
Judge calibration from outcomes. Needs 3 and 4 first, and needs volume, so it is last — but it is the reason to do 3 early.
Konvert or Foreplay as market input, per creative-intelligence.md, once coverage is verified.

Sources

Judge reliability: LLM-as-a-judge: how it works, when it fails, why frontier models fail 50%+ of bias tests, position bias in rubric-based judging, pairwise comparison evals · Debate: multi-agent debate survey and taxonomy, when helping hurts, and how to fix it, minority sentinel: overturning majority voting · Attribution: incrementality testing methods, creative testing methods · Strategy versus execution and pricing: what to look for in a performance creative agency, agency pricing by service