We fed our AI raw knowledge, and truth got worse. So we moved to checked facts.

The obvious way to make an AI write truthfully about a business is to give it the business's documents. Index them, search them, paste the best chunks into the prompt. This is retrieval-augmented generation, or RAG. It is the standard answer.

In our tests it made things slightly worse. This is how we found out, and what we replaced it with.

Why we did it

YuktiBOS writes for clients. A post that invents a statistic or a company history is worse than no post. So truth was never optional.

Our setup used RAG. Client knowledge was split into chunks, embedded, and the top six chunks above a similarity floor of 0.15 were pasted into the writer's prompt. That floor is low. It lets in a lot of loosely related text.

How we tested it

The Content Lab has an "ablation" variant called "No client knowledge (RAG)". It runs the same brief with the knowledge search switched off. A plain model baseline is a different thing: it also drops the brief and the brand card.

That distinction matters, and it nearly fooled us.

First result: the automated scorer said RAG hurts truth

In one lab run (Lab 2C, Mukika), the no-RAG variant on posts scored an automated overall of 5.7 and Truth of 7.2, with 100% rated usable (n = 3). Against it, posts in the main runs had automated Truth scores between 3.4 and 5.4. In my planning notes I wrote that loose RAG hurts post truth, "3.4 to 7.2 without it".

The sample was three pieces. I did not trust it yet.

Second result: careful readers, much bigger sample

After we had a full blind read of 284 pieces (the previous post), I could compare properly.

A naive cut looks good for RAG. Pieces with client knowledge averaged 4.7 against 3.8 without, and invented-truth rates were 9% against 33%.

But nearly every piece "without RAG" was either a plain model baseline or the no-RAG variant. The plain baselines are terrible for a different reason: no brief, no brand card. So the raw gap mixes in the plain-baseline effect. The cleaner comparison removes them:

So against a plain model, a brief with client knowledge is far better. Against a brief without raw knowledge search, raw RAG is neutral to slightly worse. On posts it was 5.3 against 6.0, though the no-RAG group has only 10 pieces.

The best cell was not the no-RAG variant. Human brief on Opus, with knowledge, scored 6.9 on posts with 0% invented facts (n = 18). So I did not conclude "turn RAG off". Our automated advice to make no-RAG the default was not supported either. The honest reading is that raw top-6 retrieval at a loose floor does not earn its place, and a good short brief does most of the work.

What truth failures actually looked like

The review notes show how truth fails, and it is not always "the model made it up":

• Plain baselines and Gemini invent first-person history: "a founder asked me last week", "we switched carriers twice", "50 sites we audited".
• Some company self-claims were false and recurred, which suggests a bad line in a source document, not a model slip.
• Unchecked figures: an unverified price, index percentages, vendor prices that disagreed between pieces.
• Hook variants retold an illustrative "picture this" scene as real ("In a recent session...").
• The opposite failure also happened. Drafts over-disclosed ("not found in our sources", "we haven't read past the summary") and shipped a price comparison with no prices.

None of these is fixed by pasting more text into a prompt.

What we built instead

The plan I wrote on 3 November, and the build that followed:

1. A verified fact store. Each fact has a source, a date, a verified flag and an id (F1, F2 and so on).
2. The writer gets a short human brief and only the relevant facts from the pack, and is told to use only those for any number, name or claim. Context for the writer is kept to roughly three to four thousand characters. More context lowered quality in our tests.
3. The checker maps each factual sentence to a fact id. Unsupported claims are removed or turned into opinion. An invented number blocks the piece.
4. A never-say gate in code, from the client's own list.
5. Research gathered in parallel by sub-agents (web, client facts, market context) and merged into one pack before the writer starts.
6. Hybrid search with re-ranking, a higher similarity floor and a retrieval evaluation set, still on the list for the knowledge layer.

The fact store, the never-say gate and the client-facts review are live on main. The decision recorded on 4 November: "facts instead of raw RAG."

A related idea: the full research pack for articles and the numbered-fact checks are being specified once, in one place, after a spec review found that two of our own specs described the same machine twice.

What is still open

• The "Research pack only" variant exists as a plan. I have no recorded head-to-head result for it yet.
• Cell sizes are small. The no-RAG comparison rests on 14 pieces overall and 10 posts.
• The reviewer was an AI editor, as the previous post explains.
• Hybrid search, citations, and a retrieval evaluation set are not built.
• A Mukika one-line description is still needed from me, to correct a wrong brand fact the writers kept using (a separate post).

Numbers at a glance