The low-budget instruction: two instruments, two answers

One of our test variants added a single sentence to the AI's instructions, along the lines of "the budget is low, be efficient". We wanted to know if it saved money without hurting quality.

Our automated scorer said it hurt truth and told us never to use it. Careful readers said they saw no harm. Both used the same pieces. This post is about what to do when your instruments disagree.

Why we did it

AI cost is a real constraint for us. We are a small company paying per call. If a sentence in the prompt can make the writer spend less without losing quality, that is worth knowing. We called these "low-budget twins": the same variant, same model, same brief, with and without the sentence.

I also had a hunch from my own use of chat tools. When I told a model my actual limits, it behaved more efficiently. So the behaviour might depend on how you say it.

How we tested it

The twin runs sat inside Lab 2C for Mukika: 14 low-budget pieces. Everything else in the review, 270 pieces, had no such instruction. We tracked the instruction as its own flag, separate from the variant name.

What the automated scorer said

The Lab 2C read-out was that the low-budget twin hurt Truth, so never use it. My own note on 3 November recorded the cause as: the instruction made the writer cut corners and invent facts.

What careful readers said

When we read the pieces blind (the review described in an earlier post), the picture flipped:

That raw gap of +1.3 is not evidence. The low-budget group is small and it sits inside particular runs, so other things differ. Matched pairs are fairer: same run, same variant, same model, same format, with and without the flag.

On posts, the flag is neutral to better. On articles, both pairs went down by 1.0, but each cell is one piece. I would not build anything on a single piece.

What I conclude, and what I do not

I conclude three things.

1. The two instruments disagree, and the disagreement is data. Earlier I found that the automated scorer correlates only 0.27 with careful readers, and that its Truth sub-score has almost no relationship with what readers valued (-0.11). So its warning about truth was weak evidence. That does not make the readers right. It means neither instrument has earned my trust on this question.
2. The test was badly framed. "Be efficient" is a vague instruction. A better test gives the model the real budget and says what to spend it on, such as research and facts, not filler. I have named that variant "budget-aware" and not run it yet. As far as my notes show, it has not been tested.
3. The sample is too small to settle anything. Five matched posts per cell cannot support a rule either way.

I do not conclude that low-budget instructions are safe. I conclude that the evidence we have does not support "never use it", and does not support "always use it".

How this shaped YuktiBOS

• Lab results now show the number of pieces and an "early sign" label for thin results. The plan also asks for at least five pieces per arm before anything is called "Best". That came directly from small-n problems like this one. (These fixes were built in the week of 3 to 4 November; they are described in our help docs.)
• Low-budget stays a separate flag in the data, so any later test can be matched properly.
• The design principle I now follow: if cost matters, design the budget into the system (cheap models for extraction and checks, a fixed prefix that can be cached, a cap that finishes the piece instead of stopping to ask), rather than hoping a sentence in a prompt does it.

What is still open

• The "budget-aware" re-test has not been run.
• Articles and other formats have only one matched pair each.
• The reader in these tests was an AI editor, not a person. See the earlier post.
• Absolute rupee costs for these pieces rely on an exchange rate and a price table I have not yet confirmed (see the cost post).

Numbers at a glance