The Content Lab as a tool: what it does, what it cost, and the bugs

The problem behind it

Choosing an AI model and a writing method by reading three pieces and going on feel does not scale. The Content Lab exists so the answer to "which model, which method, which prompt, for which format" comes from comparable runs, with cost beside quality.

It is for the super admin only. A run costs real money, so the page shows an estimate first.

How it works

From the help article:

1. Pick a client and fill a brief set. Starter briefs can be made from client knowledge. The core set has nine formats.
2. Press Estimate, then "Run the matrix" (models and writing variants against briefs) or "Run the full test". Optionally also run every configuration with the low-budget instruction.
3. Runs write the pieces. A scoring step marks them. You can continue a run, raise its budget, retry stopped pieces, or start more and stop starting more.
4. Read the leaderboard by variant: scorer version, overall, human score and count against the automated score, "does its job", angle, voice, average cost, cost per usable piece, cache hit and prompt size. It also says whether the client is "Ready for operations", "Not ready", or "Not enough scored pieces yet".
5. "Make this the default" for the winner, per client and format.

There are ablation variants that take one part out of the whole chain: Bare, No skills text, No lessons, No client knowledge, No quality revision, No angle step, No examples, Full voice guide, Plays on, and the same recipe with old step prompts. None has a recorded result yet.

Benchmarks are real posts read from LinkedIn, matched to the client's account size. Each client has a "Speaks as" setting: brand, founder or both.

The decisions

• A budget per run, not the workers' daily limits. Early test runs hit the provider cap and stopped everything. A run now has its own budget and skips worker caps, though provider caps still apply.
• Never wait for a person. The runner must never stop to ask. A stuck run lock now expires, and "Continue this run" touches only unfinished pieces.
• Show failures. The leaderboard used to hide them: a model with 12 failed and 4 scored pieces looked like n = 4. It now shows failed counts per group.
• Do not name a "best" from thin data. The plan is at least five pieces per arm, with the number and spread shown. The build added an "early sign" label for thin results.

The cost conversation

On 3 November the plan for a "full test" was 153 pieces for about Rs 3,388, of which scoring was Rs 1,498, or 44%. The scorer was costing nearly as much as the writing. I decided not to run it. My API credits were about $58 (a promotional $38.79 plus $19.52 of my own), and they also had to run YuktiBOS.

Instead the plan became a focused test: Mukika posts and articles only, Opus with the human brief as the base, arms with and without the offer card, a low-budget twin and three ablations, two pieces per arm, no Gemini and no best-of-3 (both had already lost), target at most Rs 800. The judge would move to Sonnet or one call.

Then the v8 judge calibration itself stopped at Rs 584 after 27 of 40 pieces, about Rs 21 a piece. That story has its own post.

The bugs

The bug list from reading the code has 25 items. Twenty-two are fixed or closed, three need my decision and four non-Lab test files still fail. A sample:

• "Score the unscored pieces" did nothing, because the scoring loop ran after the page's answer was sent, and the host throttles the CPU then.
• Failed GPT and Gemini pieces were counted as "finished": a run showed 48 finished and 16 scored.
• Long pieces never got a scorecard because the judge's reply limit was 2,800 tokens and an article's claims list got cut off and thrown away.
• Pieces parked at one provider's limit froze every piece of the run, including the Opus ones.
• Money figures may be about 14% low (rupee rate) and the Anthropic price table has no source recorded. Both need my check.

What's next

The offer-card and research-pack variants are specified, not yet run head to head. The human review found that the automated score and the readers disagree, which is why the scorer was rebuilt. I want a person, not only an AI editor, to read a sample.

Numbers at a glance