When AI writes your software, a quiet corruption sets in: the same intelligence that wrote the code also writes its tests, reads both while grading both, and — without any malice at all — starts grading its own homework. The tests pass. The software is broken. Everyone involved is being perfectly confident about it. This is the workshop's answer: the tests are built in a separate room, by someone who has never seen the code.
The rule: tests come after, from outside
Conventional wisdom says write tests first or alongside. For human teams that works because humans have independent judgment about what "correct" means. For a single model doing both jobs, test-after-independent is the only order that works:
- After — the spec is finished first, so tests check against intent, not implementation details. A test that mirrors the code's internal steps will happily verify a wrong algorithm done consistently.
- Independently — a separate agent writes the test suite from the spec alone. It never reads the implementation. It can't, physically.
Blindness, enforced with chmod
"Keep them separated" is a suggestion; operating systems offer enforcement. The implementation tree gets its permissions locked away from the test-writer's context before test generation starts. The test agent isn't trusted to not peek — it's unable to. This is the whole trick of the methodology: replace "please don't contaminate yourself" with a filesystem that makes contamination impossible.
Split roles: searcher vs judge
Evaluation splits across two agents as well. The searcher hunts for failures — adversarial inputs, edge cases, hostile sequences. The evaluator judges whether observed behavior matches the spec. One agent doing both tends toward leniency (it understands why the code did the weird thing, and understanding is seductive). Split agents argue, and the argument is where honest verdicts live. It's the same principle as the parallel memory systems: single points of confident judgment always drift.
The payoff: gates you can actually trust
This rig plugs into the phased-gate pipeline — integrity checks, reproducibility checks, transfer checks — and changes their character entirely. A green gate now means something specific: an agent that never saw the code, testing from intent alone, judged by an agent incentivized to disagree, could not break it. That's a claim about reality. A self-tested system's green gate is a claim about the model's mood that afternoon.
// the closing loop
The gate infrastructure itself lives in a git repository guarded by its own commit gate and an hourly watchdog that flags ungated edits. Infrastructure that watches itself sounds like a joke about recursion until you realize it's just the same rule applied one level down: nothing grades its own homework, including the graders.
Cheaper than it sounds
The objection is cost: two extra agents per evaluation. The rebuttal is arithmetic — one silent wrong merge that reaches production costs more debugging time than a year of independent test suites, and debugging your own confidently-wrong output is the most expensive activity in software. Blindness is nearly free. Confidence is what costs.