My Best Reviewer Never Saw the Code

When an AI agent builds something, the usual check is a set of tests that the same agent wrote. This week I added a second judge to a small project: a separate AI that never sees the code, only the finished thing, the way a user would. The tests passed every round. The judge's scores went down before they came back.

The project was a game. I've had agents working on an ocean simulation for a few weeks, much of that time trying to make it look like a photograph. It never got there. In a blind test, a separate AI model was shown one real photo and one render at a time, and it picked the real photo 36 times out of 36. What the simulation does have is a 46-foot sailboat you can steer, an island shaped from real elevation data, and weather from flat calm to storm. It looked good and there was nothing to do in it.

So I asked for a game instead: Open Sea: Squall Run. You sail between six pairs of buoys off the coast of an island while a squall builds, and the course closes at a fixed time. It runs in a desktop browser and needs a keyboard.

Two kinds of check

One agent built it: the rules, the course, the controls, the screens and the sound. It also wrote a scripted test that plays the whole thing. It starts at the title screen, sails every gate, reaches the results, retries, loses the race in each of the three possible ways, and fails on any error.

That test has a blind spot I didn't see at first. Its autopilot steers by reading the game's internal data, so it always knows exactly where the next gate is. A player only has the screen.

So each round, a fresh AI agent gets the game and nothing else: no code, no design notes, no memory of earlier rounds. A small program lets it take screenshots, read the text on screen and press keys. Its brief is to work out the rules from the screen, finish a race inside ten minutes, and score four things from 1 to 5: how clear the game is, how the boat feels, how fair the challenge is, and whether it wants to play again. I wrote the bar down before any of the game existed: 4 or better on all four, two rounds running.

The scores went down

Round one scored clarity 4, feel 3, challenge 3 and replay 3. The critic finished on its first try in 4:40, which was the problem. The title screen teaches one real skill, zig-zagging to make progress into the wind, and the critic never needed it, because every gate could be sailed straight at. It also caught that the storm I promised never actually ended the race, and that hint boxes covered the instruments on a laptop-sized screen.

The building agent fixed all of that. Two legs now point straight into the wind, the time limit is real and shown on the clock, the boat is heavier, and shrinking the sail in a storm is safer but visibly slower.

Round two, with a new critic, scored clarity 3, feel 2, challenge 2 and replay 2.

This one missed the ten-minute budget by about 30 seconds. It couldn't see where the line between two buoys was, so it sailed past the finish twice without either pass counting. It capsized while doing what the on-screen warning told it to do. In the storm the warning flickered on and off every second or so, and sailing turned into tapping the space bar on cue. Most of those problems came from, or were exposed by, the round-one fixes. The scripted test passed both versions without a complaint.

Round three draws a glowing line between each pair of buoys, warns on every missed gate, warns before shallow water, steadies the storm warning and makes the capsize advice match what actually saves the boat. The third critic finished its first race inside the budget, in 5:54, and scored clarity 4, feel 3, challenge 3 and replay 3. That's back to round one's numbers, on a course that now needs the skill it teaches. Its main complaint is about that skill: a turn through the wind stalls the boat too often, and getting upwind is the least fun part of the race. The bar isn't met, and round four starts there.

Two caveats. The critic is an AI looking at screenshots, so it reacts a beat late, and some of its mistakes are that lag rather than the game. And each round is one run by a different critic, so a single point either way is partly noise. I still trust the direction. Nobody but agents has played the game yet.

What I'd tell a team using agents to build things people use

Separate the builder from the judge, and keep the judge away from the code. A reviewer who has read the code knows what everything was meant to do. One who only has the product finds out the way a user would.

Write the bar down before the work starts, and make it about people. "The tests pass" was true from the first hour. "Someone new finishes in ten minutes and wants another go" is still not true, and that's the bar that matters.

Expect a score to drop after you fix exactly what you were told to fix. Fixes interact, and a reviewer with no memory of the last round only sees what you have now, not how hard you worked.

Keep the scripted tests too. They caught real breakage every round. The two checks answer different questions, and the scripted one is the cheap one.

The next step is the one no agent can take for me: putting it in front of people.

Credits: island terrain produced using Copernicus WorldDEM-30 © DLR e.V. 2010-2014 and © Airbus Defence and Space GmbH 2014-2018 provided under COPERNICUS by the European Union and ESA; all rights reserved. Sky photographs from Poly Haven (CC0). Built with three.js.