An evolution harness tunes strategies by letting them compete: vary the knobs, run the tournament, keep the winners, repeat. Ours tuned a real workload — and kept producing champions who couldn't play. The final scores were beautiful. The actual performance was a coin flip. This is the story of why, told without a single statistics term, because the whole problem fits in one sentence: a tournament with noise in it is a lottery, and optimizers are excellent at entering lotteries.
The practice-vs-game gap
Every generation, the harness crowned a new best strategy by how it scored during selection. Every deployment, the champion underperformed that score against fresh opponents. The pattern was consistent enough to stop being coincidence: the optimizer wasn't finding the best strategy. It was finding the strategy that got luckiest during tryouts.
Think of it as hiring. If you interview fifty candidates once each and hire whoever had the best single day, you'll mostly hire people who had one good day. Some of them are genuinely great — and some just rolled well when it mattered. You can't tell which from one day. Neither could the tournament.
The scoreboard was lying politely
Two measurement sins made the lottery worse. First, the evaluation environment drifted from reality in small ways — practice conditions that didn't quite match game conditions, so strategies specialized for the exam instead of the sport. Second, the noise floor was enormous relative to real skill differences: rerunning the same strategy twice could swing its score by hundreds of points. When your measurement wobbles more than the thing you're measuring, your rankings are astrology with extra compute.
// the fix that actually fixed it
Champions must defend the title. Before any winner gets crowned, it rematches the previous champion enough times that luck washes out — five-plus head-to-head rounds minimum, because the observed noise demanded it. One lucky fool can win a game. It takes genuine skill to win five rematches.
What changed after demanding rematches
- A probe budget that sounded insane and was merely correct. An early config ran 128 probes; analysis said the noise floor needed roughly eight times that to separate signal from luck at the top. Adopting 1000 probes felt wasteful right up until the first rematch cycle rejected a "champion" that would have shipped broken.
- A replay shield. Past winners periodically re-run against the current field. If yesterday's champion beats today's, the optimization went backwards somewhere, and everyone needs to know that immediately.
- Adversarial probes aimed at risk, not sprinkled everywhere. Extra hostile test cases target exactly the steps where being wrong is expensive — cheap insurance where insurance is due, no tax on safe steps.
- Fewer knobs. A 14-parameter tuning space produced configurations that transferred to the real game only a third of the time. Cutting to fewer, saner knobs raised transfer dramatically. The optimizer's job isn't to use the search space; it's to find things that survive contact with reality.
The plain-language summary
If the same thing measured twice gives different answers, then any single measurement is a lottery ticket. Optimizers exploit noise faster than humans can see it, so they will always, always enter the lottery unless the rules force rematches. Practice great, play worse isn't a model failure — it's what happens when selection happens on a noisy stage and deployment happens on a real one. Make the stage real, make the winners rematch, and the lucky fools stop getting hired.