Evals Are the Only Moat

A moat is the water around a castle. It is the thing that makes the castle hard to take. In business people use the word for whatever advantage a competitor cannot simply copy.

Now look at an AI product and ask what its moat is. Your prompts? Someone can work those out by asking your product enough questions. Your document pipeline? Someone can rebuild it from a blog post in an afternoon. Your model? Someone switches to the same one by changing a single line of configuration.

Every visible part of an AI product is, to a decent engineer with a free weekend, a copying exercise.

The part that does not copy is your answer to a boring question: did that change make it better? Not "does the demo still work" — better, measurably, on the cases you actually care about. Teams who can answer that improve every week. Teams who cannot are performing improvement, and the gap between those two grows quietly until it is enormous.

The tests that answer it are called evals. This is an argument that they are the whole castle.

Why your normal testing instinct fails here

In ordinary software, correctness is local and it is yes-or-no. The function returns 7 or it does not. You write a test, it passes forever, and your test suite becomes a ratchet — a thing that only turns one way. Behaviour that works today cannot silently stop working tomorrow.

AI breaks that in two ways at once.

The output changes every time. Ask the same question twice, get two different sentences. So a test passing once proves less than it used to.

Correctness has degrees. An answer can be accurate but so terse it is unhelpful. Or helpful but subtly wrong in a way that only matters to one kind of customer. "Is this string equal to that string" has nothing useful to say about either.

So people do the reasonable thing: they test the parts that are yes-or-no. Does the JSON parse? Did the right function get called? Does the shape match? And they check everything else by looking at it.

This is where the trap closes. All those checks keep passing while the product gets worse, because none of them is measuring the thing anyone cares about.

What building an agent taught me

I built WebGenie, a browser agent that clicks around websites to do tasks for you. It runs a loop of three roles.

A planner decides the next step. A navigator carries it out on the actual page. A validator decides whether that step achieved what the plan meant.

The validator exists because the navigator lies. Not on purpose — but "I clicked the button" and "the thing the button was supposed to do actually happened" are two different claims, and only the second one matters.

That third role turned out to be the most valuable thing in the project, and it took me embarrassingly long to see why. The validator is a test harness that happens to run in production. It is the piece that converts "the agent did something" into "the agent did the right thing," and every improvement I made to the agent came from first improving the validator's ability to tell those apart.

When I finally made the validator strict, the agent's real failure rate stopped being a feeling and became a number. The number was considerably worse than my impression had been.

That is the normal outcome. It is also the point.

Start with thirty rows

Most teams do not have evals because "build an evaluation suite" sounds like a quarter of work. It is not. The first useful version is thirty examples and a script, and you can have it before lunch.

Take thirty real inputs — from your logs if you have them, from your own hands if you do not. For each, write down what a correct answer must contain, in whatever form you can check: an exact phrase, a set of facts, a specific function call, a short checklist. Run them. Count.

That number is now the only opinion in the room allowed to settle an argument.

Thirty is not statistically impressive and does not need to be. It needs to catch the case where a prompt tweak that fixed one thing quietly broke six others — and thirty examples catch that every time, because these failures are not subtle once somebody is actually looking.

Three ways to grade, best first

Let a program check it. Was the right tool called with the right inputs? Does the generated command produce the intended files when run somewhere safe? Does the extracted data contain the right values? Anything a program can check, let a program check. It is exact, instant and free — and far more of your system is checkable this way than feels true at first.

Compare against a known-good answer. Not word-for-word — a checklist of facts the answer has to include, and facts it must not.

Ask a stronger AI to judge it. This is the flexible one and the one to distrust most. It leans toward answers that are long, confident and nicely formatted. It will happily rate a fluent wrong answer above a blunt right one.

If you use it, calibrate it first: grade fifty answers by hand, grade the same fifty with the AI, and see how often they agree. If they often disagree, your instructions to the judge are the problem — and you have learned that before trusting a single automated score.

The examples worth keeping

The most valuable test cases are not the typical ones. They are the ones that already broke.

Every time somebody reports a bad answer, every time you spot a wrong output — that case goes into the suite with the correct answer written beside it. Two minutes of work, and it does something no amount of invented test data can: it guarantees that one specific, real, observed failure can never come back unnoticed.

Do that for six months and you own something describing your product's actual failure surface. Nobody can copy that from the outside, because you only got it by having real users hit real problems.

That is the moat, stated plainly. Not the prompts. The accumulated, verified record of every way your system has been wrong.

What I would do on day one

• Thirty cases before any prompt tuning. Not after.
• Measure "did we find the right information" separately from "was the answer good."
• Let programs check whatever programs can check; use an AI judge only for the rest, and calibrate it.
• Every reported failure becomes a permanent test case, the same day.
• Run the suite on every change, and let the number end the argument.

Models will keep getting better, and every improvement lands for your competitors on the same morning it lands for you. The advantage that compounds is not the model. It is knowing, faster than anyone else, which of your own changes were real.