We graded 17 AI deliverables. Two got a B-minus.

On 24 October I reviewed the work our AI engine had marked "Done" and did not like what I saw. This is the story of what we found when we looked properly.

What I saw

Three tasks stood out.

• "Research launch venues and PR contacts" came back with invented US venues, priced in dollars, for an Indian company. It also called Mukika a "functional beverage" brand. The task was marked Done while its review was still blocked.
• "Draft logo concept directions" was Done. There were no brand guidelines and no review.
• "Fact-check messaging sheets" did not check the sheets. The plan drafted new content instead.

I asked for a diagnosis, not a patch. A review agent read the code and graded a sample of 17 finished AI deliverables. The result: two B-minus, eight C, six D and one F. Every list, event and form deliverable was D or worse.

What was actually wrong

The diagnosis ranked eight root causes. The ones I found most useful:

1. The kind of work did not shape the steps. Everything went through blog and SEO machinery. A venue list got an intake that asked for an SEO keyword, a writer that used keyword research, and a reviewer that checked brand, SEO and accuracy.
2. Client facts did not reach the steps. Before 17 October there was almost no context. After it, the facts block was cut at 12,000 characters in a fixed order, which dropped Mukika's brand principles, design system, website and location. "Attached memory" was displayed but never read.
3. Nothing grounded the work. Most steps had no web search and nothing said "do not invent". Research could finish with zero sources. A made-up URL passed the source check.
4. Reviewers never saw the task or the brief. In August they got an 80-character summary instead of the draft.
5. "Done" did not mean approved. A task closed when its workflow ended. Five saved recipes (logo design, competitor research, campaign plan, basic blog, monthly report) had no human gate at all.
6. The flow page lied. It showed stale "running" steps from August and summed costs.
7. Output was free text that no code checked.
8. There was no "redo with my notes".

What worked, said the diagnosis, was Market Finder and the Marketing Kit: a compact client picture built in code, web searches in code, "to verify" on guesses, and checked JSON replies. Reuse that.

The proposed fix, in three phases

Phase 1 (three to four days, no schema change): turn off the five gate-less recipes; add a kill switch so the dispatcher auto-runs only allow-listed kinds (start with social posts); close a task only on approval; list falsely-done tasks so I can confirm and reopen them; banner drafts made before 17 October; fix the facts cut.

Phase 2 (two to three weeks): rebuild per-step context, research with sources, reviewers with a rubric per kind, truthful completion, and redo with notes.

Phase 3: no kind of work auto-runs until its recipe has been reviewed and passed a benchmark.

A side branch with a document-review kind, a composer guard, and "Reject asks why" (so reasons teach the composer) was built but not deployed.

What I learned

The part that stays with me is the shape of the failure. Nothing in this engine was lazy. Each piece worked for its original job. The mistake was routing every job through one machine and trusting "Done" to mean done.

Two later decisions come straight from this. First, "Done needs a document": a task is closed only when a deliverable exists (ADR-416). Second, a client ticket starts work automatically but passes one human review, with no hidden second gate.

What's still open

The graded sample is 17 pieces, graded by the review agent, not by a panel of people. I have not re-graded a comparable sample since. The later Content Lab work gives much better evidence for content, but the non-content kinds (lists, events, forms) have not been re-measured.

Numbers at a glance