Evals
Evals are the practice of building a task-specific set of test cases with automatic or semi-automatic scoring, then running it repeatedly to measure whether a model-based system does the job it was built for. Where a public benchmark asks how good a model is in general, an eval asks whether this prompt, this retrieval setup and this model are good enough at this product's particular task, and whether yesterday's change made things better or worse.
An eval set is the unit test suite of an AI application, built from the application's own inputs. Each case pairs an input with something to check against: an expected answer, a required JSON structure, a list of facts that must appear, forbidden content, or a written rubric. Good cases come from real traffic, from bug reports and from deliberately hard or adversarial inputs, and the failures discovered in production are added so that each mistake is only made once. Size matters less than representativeness: fifty well-chosen cases covering the true distribution and the known failure modes are more informative than thousands of easy or synthetic ones. The set must also be kept apart from the examples used in prompts and from any fine-tuning data, otherwise scores measure memorization.
Regression evals in continuous integration turn quality into a gate. Because prompt tweaks, model upgrades and retrieval changes all shift behaviour in unpredictable ways, teams run the eval suite automatically on every change and compare against the stored baseline, failing the build when the score falls below a threshold or when specific must-pass cases break. OpenAI released its open-source Evals framework in March 2023 alongside GPT-4, and similar harnesses now exist in most LLM tooling. Since outputs vary from run to run, a robust setup uses fixed seeds or low temperature where possible, repeats stochastic cases, and reports differences with an awareness of noise: a two-point change on forty cases is often just chance.
LLM-as-judge makes open-ended outputs scorable but needs calibration. Many qualities, such as whether a summary is faithful or a reply is polite, resist exact-match checks, so a second model grades outputs against a rubric or compares two candidates. Zheng et al. in 2023, in the MT-Bench and Chatbot Arena paper, found that strong judges agreed with human preferences at roughly the rate humans agree with each other, while also documenting biases: preference for the answer in a particular position, for longer answers, and for answers written by the judge's own model family. Mitigations include swapping order and averaging, pointing the judge at a specific rubric rather than a vague notion of quality, requiring reasoning before the grade, and periodically checking the judge against human labels.
Evals differ from benchmarks and from broader evaluation suites in purpose and ownership. Benchmarks such as MMLU or HumanEval compare models on shared tasks, are public, and are prone to contamination and saturation; an evaluation suite in the broad sense may bundle many such benchmarks to profile a model. Evals in the practitioner sense are private, narrow and owned by the product team, and they answer the question that matters for a launch. A strong benchmark score says little about whether a customer-support bot handles refunds correctly.
Common pitfalls follow Goodhart's law: once a metric is the target, it stops being a good measure. Tuning prompts until an eval passes can overfit to its few cases; an automated metric may reward verbose answers; averaged scores hide a rare but catastrophic failure class, which deserves its own must-pass cases; and a stale set stops matching users as their behaviour drifts. Evals should be refreshed from production samples, sliced by category, and read alongside user feedback, since a green dashboard measures only what the cases cover.
Read evals as the memory of a team's mistakes encoded as tests: they make improvements provable and regressions visible, but only as far as the cases honestly resemble real use.