The judge that cost Rs 21 a piece, and a correlation I don't trust yet

After the blind review showed our automated scorer agreed with careful readers at only 0.27, we rebuilt it. This post covers the part where the rebuild nearly cost more than it was worth, and a number from the rebuild that I refuse to call a result.

What the rebuild did

Scorecard v8 (ADR-470 to 476) made four changes:

• Truth comes from code, not opinion. One model call lists each factual claim and the fact it rests on. Code verifies that the fact exists and that every number matches. Code also finds never-say framing, leaked notes, invented figures, quotes, named sources and first-person history. Hard breaches cap the score and force a No.
• The judge scores what the human scored: does it do its job for its format (0 to 10), client angle (0 to 2), voice (0 to 2), and publishable (Yes, Light edit or No). Six to nine real reference pieces per format, with the human's numbers, are shown to the judge. Market benchmarks are no longer shown.
• Two passes, a third on disagreement. Each piece is judged twice with the prompt arranged differently. If overall scores differ by more than 1.5, a third pass runs and each part takes the median. A reliability flag shows how steady the score is.
• The final call is the more cautious of the judge's and the formula's.

The formula for the overall score was fitted to the human numbers.

The number I won't call a result

The offline check ran v8's code and gates over 242 pieces with a stand-in judge: the human's own numbers. It gave a correlation of 0.92 and usable-or-not agreement of 88%, against 0.27 and 66% for the old scorer. The formula's fit was 0.957.

Those figures say the plumbing is right. They do not say a real model judge agrees with people. A stand-in that reads the human scores will agree with the human scores. The ADR itself says "no model is called". The real test is the live calibration: re-score a fixed 40-piece sample with v8, show the agreement, and compare with a cheaper judge. The target is a correlation of 0.6 or better and usable-or-not of 80% or better.

The cost

I pressed the button. The run stopped at Rs 584 after 27 of the 40 pieces, about Rs 21 a piece. The review of why was quick:

1. The judge ran through the same call path as the writers, so every pass carried the writer's whole context: brand card, client facts, direction pack, agent prompt.
2. Both the judge passes and the claims check used the most expensive model by default.
3. Nothing was cached.

The fix (ADR-501)

• Judge passes on Sonnet, claims check on Haiku, unless a setting says otherwise. A "cheaper judge" tier uses the small model for both.
• A lean prompt. The rubric, the "does its job" line and the reference pieces are the same for every piece of a format, so they became the system text and are cached. The user message holds only the client, short fact excerpts (2,500 characters), the offer (1,200), the brief (600) and the piece (6,000, start and end if long).
• The estimate now prices a call as cached read plus new input plus output, and replaces guesses with the average of real lean calls once there are at least three.
• Resumable: "Continue the sample" scores only unscored pieces. The cap is counted from the round's start, so a stop at Rs 584 leaves room. Agreement is shown "so far", with the judge model for each piece.

The new estimate is about Rs 2.1 a piece on Sonnet and Haiku. The same lean prompt on Opus is about Rs 6. The target was under Rs 4.

What I learned

A judge is a product with a cost. The full pipeline, the 40-piece calibration and the leaderboard ran together; a design cost target should have been in the first version. Also: do not quote the first correlation that looks good. The honest one is the one a stand-in cannot give.

What's still open

The live agreement has not been reported in what I have. When it is, I will publish it whatever the number. Also unresolved: the human labels came from one AI editor, so even a high agreement is agreement with that editor.

Numbers at a glance