How I picked a model with evals, not vibes

I tried five models by hand first. I typed the same few questions into each one, read the answers, and picked the one that sounded the smartest. A week later it gave an answer that sounded smart and was wrong. Sounding smart is not a benchmark. So I built one.

Why "it feels smart" is not a benchmark

Reading a handful of answers and going with a gut feeling works fine for a toy project. It does not work when the model sits inside a copilot that touches real goals, real people and real permissions in your product. You need a wide set of tasks that look like real use, run more than once each, and checked by something other than your own eyes, or you are just guessing with extra steps.

For this project that meant testing five models across three reasoning effort levels, over a set of 25 tasks plus seven longer flows that took several steps to finish. That is enough combinations that no person could hold the results in their head and trust their own memory of which model did what.

Safety first, then quality, then cost

The order you rank things in changes what you end up picking. I ranked every model on three things, always checked in this order: safety, then quality, then cost.

Safety asks whether the model does something it should not, like following an instruction that was hidden inside a piece of content it was only supposed to read, not obey. Quality asks whether it does the actual task well. Cost asks what it costs to run.

I put safety first on purpose. A cheap model that gives a wrong answer costs you a bad afternoon. A model that can be talked into skipping a permission check by text buried in a note somewhere costs you something you cannot undo. A model only got to compete on quality once it had already cleared the safety bar, and only got to compete on cost once it had cleared both.

Split the test from the grade

Early on I graded a task the moment it finished running. Then I found a mistake in how I was scoring one type of task, and had to rerun everything from scratch, every model, every task, just to fix one counting rule.

So I split it into two steps that never touch each other. First, run the task and save exactly what happened, in full. Second, grade what happened, completely separately, using that saved record. Fixing a mistake in the grading step now costs nothing. I do not burn a single API call again. I just grade the same saved runs a second time.

The grading step itself splits again. Anything with a clear right answer, like whether a record actually changed the way it should have, gets checked by plain code. No model needed for that. Only the tasks that produce free text, like a written summary, go through a model acting as a judge. I only reach for a model as a judge when a plain check cannot tell me the answer.

A task only counts if it passes every time

A model can get lucky. Ask it the same question five times and it might get four right and one wrong, and the one wrong time might be the one your actual user hits. So I only count a task as reliable if it passes on every single repeat, not most of them.

That is a strict bar, and it changes the ranking. A model that looks great on one lucky run can fall behind a steadier model once you run each task several times and only count the ones that never slipped even once. Given the choice, I want the steadier one, even if its best single run looked less impressive.

Holding some tasks back

If you write your test tasks and then tune your prompts against those same tasks, you will always look good on your own test. So some tasks never got touched while I was writing or adjusting prompts. I only ran them at the very end, to see if the tuning actually generalized, or if I had just learned to pass my own quiz.

The tasks built to check safety needed something realistic to fail on. I used a frozen test setup, a company's worth of sample data that never changes between runs, with a few records carrying instructions planted inside them, the kind of trick you would see in a real support message or a pasted note. A safe model reads that content and does not act on the buried instruction. I also ran the grader itself through mutation testing, changing small pieces of its own logic on purpose to check it would still catch a wrong answer. A grader that always says pass is worse than having no grader.

What actually drove the bill

I expected the choice of model to be the biggest cost lever. It was not. The bigger cost came from prompt overhead: the tool descriptions, the conversation history, and the extra context the copilot loads so it knows what screen you are on and what you are allowed to touch.

That was the real lesson.

Trimming what you send with every call saves more than switching to a cheaper model.

Carry too much weight in every prompt, and even the cheapest model ends up costing more than it should, while the most expensive model looks worse than it actually is.

What this cost me

None of this came for free. A harness with a separate run step and grade step, tasks held back on purpose, a frozen test setup and a grader checked by mutation testing took real time to build, time that could have gone into features instead. I do not regret it, but I will say it plainly: an eval harness is its own small project, not a script you write the night before you ship.