The New Prompt Felt Better — But Was That an Improvement or Just Luck?

You have a prompt that returns mediocre results. You sit down and edit it: rewrite the opener, add an example, drop a clunky constraint, and rework how you ask for the format too. Run it again. The output is clearly better. You nod, save the new version, move on.

But pause a second: what did you just learn? Four changes, one better result. Which of the four made the difference? Or is it better only because the agent happened to answer well this time, and the old version rerun would have been good too? You don't know. And next time you hit a different prompt, you carry no lesson forward — because no clean lesson was ever extracted.

01Two things break the experiment: too many variables, and the wrong baseline

To see why "it looks better" isn't enough, remember one thing about the agent: it isn't deterministic. The same prompt, the same input, run twice can produce two qualitatively different results — tighter here, more off there. There's a baseline layer of noise in every run. A real improvement is a signal you have to pull out of that noise, and noise is very good at impersonating signal.

Two mistakes guarantee you never pull it out:

The core difference: the left column gives you a feeling, the right gives you a cause. A feeling doesn't repeat and can't be taught to anyone else; a cause can. "Dropping this line makes the output tighter" is something you reuse forever — "not sure why that one came out better" evaporates the moment you close the tab.

02The one-variable-at-a-time loop

The method is borrowed straight from the lab, but far lighter — you don't need any fancy measurement, just the discipline of four beats:

Beat three is the most skipped, and the one that keeps the whole experiment meaningful. "Same input" sounds obvious, but people unconsciously change the input between attempts every time — and once the input differs, every comparison after it is void. Keeping one fixed set of inputs to test against is the smallest job with the biggest payoff.

The "one variable" doesn't mean you're condemned to inch forward forever — it means when you want to know what works, you isolate it. While exploring freely, go ahead and change a whole batch for speed. But the moment some version "seems better" and you want to lock it in as a real improvement, that's when you back up, change them one at a time, to learn which change the credit truly belongs to.

03What compounds is the real reward

The reward of this approach isn't a better prompt for today — changing everything at random and getting lucky gives you that once too. The reward is what you carry into the next time. Each one-variable test yields a small, clean, true piece of knowledge: "adding a concrete example helps this kind of output more than adding an abstract constraint." That piece isn't tied to one prompt — it's a principle you can apply to the next prompt, and the one after that.

That's the difference between someone who's tuned prompts for six months and is still guessing, and someone who's tuned for six months and now has a trustworthy set of intuitions. Both spent the same time. The first accumulated a pile of disconnected prompt versions with no idea why any of them is good; the second accumulated a set of causes — and causes are the only thing that multiplies.

A controlled test is also the natural gate before a prompt earns a place as a template: you only promote what you've known to be better, not what you felt was better. So next time you change a prompt and it looks better, don't celebrate and save it. Ask one cold question: how many things did I change, and have I rerun the old version yet? If not, what you're holding might just be a lucky run wearing the costume of an improvement.