284 pieces read blind: our automated judge agreed only weakly (r = 0.27)

We built an automated judge to score every piece of AI-written marketing in YuktiBOS. It gave numbers to the second decimal. Before I trusted any of them, I wanted to know one thing: does the judge agree with a careful reader?

It mostly does not.

Why we did it

The Content Lab (see the first post in this series) produced hundreds of pieces. We were choosing models and methods from the judge's leaderboard. If the judge is wrong, every choice made from it is wrong too.

The judge's own leaderboard had told us things like "the saved pipeline is the most usable" and "conversion is the weakest score everywhere." I wanted to test those claims against someone reading the actual text.

How we did it

First, an honest note about the reviewer. The plan in our docs says "Opus reads each one". The review files are headed "human-style editorial read, blind". So the reviewer here was a careful AI editor, Claude Opus, following a written rubric, not a panel of people. That matters. I come back to it below.

The rubric (doc 118) asks, for each piece:

• Would I publish it? Yes, Yes after a light edit (under five minutes), or No.
• Does it do its job for its format?
• How unmistakably is it Mukika? Scored 0 to 2.
• Truth: any claim not backed by a known fact. Any invented number, company or quote is an automatic No.
• Voice, 0 to 2.
• Best line, worst line, and the one change that would fix it.

Pieces were shown by blind code, with the variant and model hidden, in batches of about 20 by format. We joined the results back to the key afterwards.

The coverage

• The key has 363 rows. 284 were reviewed in five batches. 79 were not.
• All 284 codes matched the key.
• Verdicts: Yes 13, Light edit 108, No 163.
• 225 of the 284 also had an automated score. The rest were never scored (many were stuck from the bugs described elsewhere).

The results, with the maths

Agreement. Pearson correlation between the human score and the automated overall score: 0.27 (n = 225). Spearman: 0.25. I re-ran Pearson from the merged file and got 0.266, which rounds to the same figure.

Agreement on "usable or not": 149 of 225, which is 66%. A coin toss gets you 50%.

By format:

Average human score: 4.76. Average automated score: 4.41.

What the judge's parts tracked. Correlation of the human score with the judge's sub-scores:

• Craft: 0.66
• Engagement: 0.55
• Conversion: -0.04
• Truth: -0.11

So the parts of the judge that look at how the writing reads and how it might perform were fine. The parts that judge whether it sells, and whether it is true, had no relationship with what a careful reader thought.

Where it was far off. In 112 of 225 pieces the two scores differed by two points or more.

• The judge was too generous on 42 pieces. 36 of those 42 sold Mukika as a board game, which is wrong (a separate post covers this). The judge cannot see what the company is, so it cannot see the offer is wrong.
• The judge was too harsh on 70 pieces. These were clean, specific Opus posts and articles that never mentioned Mukika. One piece that reads like a telecom blog scored 3.2 from the judge and 8 from the reader.

What the review changed in our decisions

Model averages across all formats: Opus 5.5 (n = 133), Sonnet 5.4 (n = 48), GPT-5.5 3.6 (n = 56), Gemini 2.0 (n = 42). Gemini invented facts in 48% of its pieces.

We now write posts with the human brief, articles with the editor's brief with best-of-3, on Opus, and we do not use Gemini to write.

Rebuilding the judge

The weak agreement became the brief for scorecard v8. As described in our help docs:

• Truth now comes from code: claims are matched to the client's numbered facts, the never-say list is applied, leaked notes and invented figures or quotes are caught.
• The judge scores whether the piece does its job for its format (0 to 10), how clearly it is the client (0 to 2), how it sounds (0 to 2), and whether it can be published.
• It uses real reference pieces scored by the reader.
• It is asked twice with the prompt arranged differently, and a third time if the two answers differ by more than 1.5. A reliability flag says how steady the score is.
• The target is agreement of 0.6 or better.

I have to be plain here: a 40-piece calibration run against the human labels is built, but I do not have its result in my notes yet. I will publish it when it comes in, whichever way it goes.

What is still open

• The reviewer was one AI editor with a stated stance. One rule in that stance: any piece calling Mukika a board game scores at most 3. That makes the board-game finding partly the rubric's own judgement, and I say so in that post.
• Opus is also the best-scoring writer. An Opus reader may favour Opus prose. A second reader, ideally a person, would test that.
• 79 pieces were not read. Cell sizes are small in places (many cells have n under 10).
• The judge sees prose. It cannot see leaked notes, raw redirect links or alt text that describes the wrong scene. Code checks now cover some of this.

Numbers at a glance