Evals are a private, team-generated set of test cases, each pairing a specific input with its desired output. They are created before making changes to a system, serving as a direct measure of whether a prompt, model, or harness alteration truly improves performance rather than just shifting it. Unlike public benchmarks, evals are tailored to the system's actual failure modes, making them resistant to accidental training or overfitting. They demonstrate that a change made things better, not just different.
The practical value of evals lies in their role as a compass for progress. Without a fixed, internal set of cases that represent the system's actual weak points, every change becomes a shot in the dark; an improvement in one area might silently degrade another, or merely chase a metric on an irrelevant public dataset. Relying on evals ensures that iterative development directly addresses known pain points and prevents regressions on critical functionality.
A common misconception is that evals are merely a smaller, private version of a benchmark, or that their creation can be delegated to an automated "evaluation module". Instead, evals represent the human understanding of what "good" looks like for a specific task and system, informed by real user interactions and observed failures. Their value comes from being handcrafted by the team that understands the system's operational context, distinct from a generalized benchmark or an independent validation process run by an external entity.