There is an uncomfortable moment that repeats in almost every team building with AI: someone has to decide whether the next version is ready for production. Engineering ran a couple of demos. Product tried three or four cases. One person says "it looks good" and another says "it gave me a weird answer yesterday." The decision ends up being a blend of intuition, cherry-picked demos, and a thumbs-up.

In traditional software that moment is settled by a test suite that passes or fails. In LLM products that safety net does not exist. The same input can produce different outputs on different runs, two models can be equally "correct" while answering in ways that look nothing alike, and failures are rarely loud crashes: they are answers that sound reasonable and are wrong.

The discipline filling that gap is called AI evals. It is not testing under a new name, and it is not advanced prompting. It is a new practice with its own mental model, its own tooling, and its own professional community growing fast. Right now it is the skill that most separates a team that knows how to ship AI products from one that only knows how to demo them.

This article covers what AI evals are, the three types worth knowing, how to build an evaluation dataset, how to read outputs systematically, and why a Product Manager (not only an engineer) needs this discipline to make shipping decisions backed by evidence rather than vibes.

Why traditional testing falls short with AI products

Traditional testing rests on a strong premise: given an input, there is a correct output, and everything else is a bug. That premise is what lets you write binary assertions, automate regression suites, and make a build "pass" or "fail." It is the foundation of modern QA culture.

Products that integrate LLMs break that premise from several directions at once.

They are non-deterministic. The same prompt sent twice can return two different answers. Both can be valid. One can be better than the other. The assertion output == "expected answer" stops working as a quality criterion: what matters is not whether the output matches a reference string letter for letter, but whether it satisfies certain properties (it is relevant, it is faithful to the source, it does not invent facts, it follows the requested format).

They fail silently. A classic system, when it breaks, throws an exception, drops a response, returns a 500. An LLM, when it "breaks," returns a perfectly fluent answer that says something incorrect, irrelevant, or invented. The surface still looks plausible. If nobody reads it carefully, the failure is never recorded.

The success criterion is contextual. In traditional software, "correct" usually has a stable technical definition. In AI, "correct" depends on the use case, the user, the moment. An answer can be factually correct and still inappropriate in tone. It can be useful for a novice user and unprofessional for an expert one. That demands evaluation rules that change with the product, not universal assertions.

Data changes the system. When you update a model, tune a prompt, swap the retrieval layer, or add a new tool, the product's behavior shifts, sometimes subtly, sometimes not. The idea of a "stable" release weakens. Regression stops being a discrete event and becomes a permanent possibility.

If your only instrument is a deterministic test suite, you have no way to catch what hurts most in this new world: answers that sound right but are wrong, and gradual degradations that only show up when someone looks at them in aggregate. AI evals exist precisely to cover that gap. This is also why you will hear the field called LLM evaluation or AI quality assurance: different names for the same shift in how quality gets governed.

What AI evals actually are

An eval is a reproducible way to measure whether an AI system meets a quality criterion across a set of representative cases.

All three elements matter. Reproducible: the result has to be repeatable and comparable over time, which rules out testing something by hand once and calling the ship good. Quality criterion: what you measure is defined clearly enough that two different people, looking at the same output, reach the same conclusion. Representative cases: the test set reflects the real diversity of the problem (languages, user types, edge cases, adversarial attempts), not just the three examples that worked well in the demo.

A shorter way to say it: an eval is a quality hypothesis put to the test with data.

This definition has an important practical consequence. Running evals is a layer that lives across the whole development cycle, rather than a step bolted on at the end the way traditional QA works on some teams: it runs when you are prototyping a prompt, when you switch models, when you decide to push a version to production, when you monitor live traffic. Each of those moments calls for a different kind of eval with a different goal.

That is why teams building AI products end up talking about "an evaluation suite" instead of "a test suite." The distinction is not cosmetic. It reflects a change in how quality gets governed in this new paradigm.

The three types of evals every PM should know

Evals organize into three broad layers. Each layer has a different cost, speed, and coverage surface. A mature system combines them; an immature one tends to lean on just one and carry the blind spots of the other two.

Deterministic evals

These are the cheapest and fastest layer. They cover everything you can verify without anyone's opinion: that the output is valid JSON, that it respects a maximum length, that it contains (or does not contain) certain words, that it matches a reference value exactly when the task is a closed classification.

Their big advantage is that they run in milliseconds, with no added inference cost, and they are perfectly repeatable. Their big limitation is that they only work for objective criteria. If what you want to evaluate is whether the answer "is helpful," "is respectful," or "is well explained," deterministic evals cannot reach it.

A good practice is to maximize what falls into this layer before moving to the next. The more criteria you can express as rules, the less your suite depends on probabilistic judgment, and the faster and cheaper iteration becomes.

Probabilistic evals (LLM-as-judge)

When the criterion is subjective, contextual, or hard to capture with rules, the practice consolidating across the industry is to use a model as the evaluator. This is known as LLM-as-judge. The judge receives the original input, the output of the system under evaluation, and a rubric, then issues a verdict: a categorical label ("correct / partial / incorrect"), a score on a scale, or both.

This layer is where most of the interesting decisions in AI products get evaluated: tone, faithfulness to the source, coverage, clarity, policy alignment. It is also where the next problem shows up, one many people underestimate: an unvalidated judge is an opinion dressed up as a measurement. LLMs used as evaluators carry known biases, including a preference for longer answers, sensitivity to presentation order, and anchoring on the familiar. If you do not check them against human judgment, you end up optimizing against the judge's biases instead of against real quality.

That is why a mature practice does not end at "I set up an LLM-as-judge." It ends at "I calibrated an LLM-as-judge against human annotations and I measure its agreement regularly." If you want to go deep on this specific technique, it deserves its own treatment: rubric design, categorical versus numeric judges, and the bias patterns that quietly wreck your numbers.

Human evaluation

The most expensive and slowest layer, and for certain criteria, the irreplaceable one. You use it when you are defining what "good" means in a new domain (you cannot ask an LLM to evaluate something you have not finished defining yourself), when you are validating a probabilistic judge, when the cost of failure is high, or when you are doing qualitative research to understand why your product is breaking in a particular way.

Human evaluation means something far more structured than a PM glancing at five outputs before the release. It is a process with trained annotators, clear rubrics, and inter-rater agreement metrics. Without that structure, two people acting in good faith can reach opposite conclusions about the same output, and your measurement loses its value.

The practical rule worth keeping in mind: there is no single correct type of eval. The design question is not "which one do I use?" but "which combination covers the quality dimensions of this product, within my budget of time and money?"

The golden dataset: your most valuable asset

An eval suite without a good dataset is a thermometer with no scale: it measures something, but that something does not necessarily resemble the reality you want to govern. The evaluation dataset is the set of cases you run your evals against. Some teams call it a golden dataset, others an eval set; the name varies but the piece is the same, and it is, in many ways, the most valuable asset an AI team can build.

A good dataset has three properties.

It is diverse. It covers the real dimensions of the problem: user types, languages, tones, lengths, happy paths, edge cases, adversarial attempts. If your product has to work in English, Spanish, and Portuguese, your dataset has to as well. If your users can be both novices and experts, both profiles need to be represented.

It is representative. The distribution of cases in your dataset resembles the distribution you see in reality. If 70% of real traffic is simple queries and 30% is complex, that proportion should be reflected in your dataset. A dataset made only of hard cases leads you to optimize for rare scenarios and lose sight of regressions in the main flow.

It is living. It grows and changes. When a user reports a failure, that case gets added. When a new type of question appears, you include it. When a section of the product stops existing, its associated cases retire. A dataset untouched for two years does not represent today's product; it represents the product from two years ago.

So how do you build a dataset when the product has no real traffic yet? This is where a technique few teams use well comes in: generating synthetic datasets from the dimensions of the problem. If you know the axes of variation that matter (language, user type, intent, complexity), you can generate combinations that cover the space intentionally, without waiting for real users to bring you cases. The quality of a synthetic dataset depends almost entirely on how well you mapped those dimensions beforehand.

Failure taxonomy: how to read outputs systematically

There is a skill that separates someone who understands evals from someone who only configures them: reading outputs systematically. People who come into evals from the product side tend to want to skip this step and go straight to setting up the judge. It is an expensive mistake.

When you run your system against a dataset for the first time, what you have in front of you is a soup of outputs. Some are perfect. Some are clearly wrong. Others sit in a gray zone that is hard to articulate without prior work. If you configure an LLM-as-judge before understanding what kinds of failures exist in your product, you will end up measuring invented categories instead of the real failure modes.

The mature practice is called building a failure taxonomy: manually reviewing a large enough sample of outputs (one hundred, two hundred, depends on the product) and grouping them by error type. Not "bad answers," but something like: "answers that invent dates," "answers that ignore retrieved context," "answers that switch languages midway," "answers that confuse two similar products." Each of those clusters is a failure mode.

That taxonomy is the input that feeds everything else. Judge rubrics are written against those failure modes. Regressions are monitored by category, not in aggregate. Roadmap decisions get prioritized by which type of failure hurts real users most. Without this layer, evals measure "quality" in the abstract and fail to translate findings into concrete product improvements.

One way to picture it: evals tell you how well your system is doing; the failure taxonomy tells you where it is breaking. The two questions are different and both matter.

Evals in production: continuous monitoring and regression detection

Up to here, evals live against controlled test sets. That is useful for iteration, but it falls short once the system is in the hands of real users. AI products, in production, face traffic your dataset never anticipated: unexpected prompts, jailbreak attempts, domains your team did not foresee, emergent behavior after a model change.

The evals-in-production layer handles that. It has three central pieces.

Sampling and continuous evaluation. You cannot evaluate 100% of traffic (it would be expensive and almost always unnecessary). Instead, you sample a percentage, apply your eval suite (mostly probabilistic), and watch the distribution of results over time. If aggregate quality moves, you notice.

Regression detection. When you change a prompt, a model, or a retrieval configuration, you compare the new version's performance against the previous one, on the same dataset and on real traffic. If a key metric drops below a threshold, you abort the rollout. This is the equivalent of a regression test, with the difference that the result is probabilistic: you are not verifying "this case passes," you are verifying "the distribution of results is still acceptable."

Operational guardrails. A layer that is not evaluation in the strict sense but lives nearby: inline rules that detect and block (or redirect) dangerous behavior in real time. If your agent attempts an action outside its scope, if a response crosses a toxicity threshold, if cost per interaction spikes, guardrails act before the output reaches the user.

Running evals in production is what turns the discipline from "once before the release" into "a permanent layer of the operation." It is also where a product team with good judgment can communicate quality and risk to non-technical stakeholders with quantitative backing, instead of defending itself with anecdotes. When the system you are evaluating is an agent that acts over multiple steps, this gets harder still, because you have to score the whole trajectory rather than a single answer. That is a discipline of its own, covered in how to evaluate AI agents, and it builds directly on everything in this article.

Why Product Managers need to learn evals

There is a widespread belief that evals are the technical team's problem. The reality is the opposite: evals are, before anything else, a product tool.

The reason is simple. The core question of evals is a product question, not an engineering one: what counts as a good output in this case? An engineer can implement the rubric, automate the judge, scale the suite. But the criterion for what is "good" has to come from whoever understands the user's problem, the context of use, and the cost of each type of failure. That someone, on most teams, is the PM.

Without this skill, the PM ends up in one of two uncomfortable places. Either they fully delegate the quality decision to the technical team, which hollows out their role on the most important product they are building. Or they trust intuition and demos, which works until a silent failure blows up in production and nobody on the team can explain, in any structured way, why it happened.

The transition from PM to AI PM demands several new skills, but this is the least optional of them all. Evals connect directly to the rest of the role: they are how you decide whether an opportunity surfaced in AI product discovery actually works once built, and how you keep an agentic product reliable once it can act on its own.

There is also a dual profile worth naming: QA and testing professionals. The culture of inter-rater calibration, the discipline of reading failures systematically, and the practice of reporting with rigor are skills that transfer almost directly into AI Quality. The difference is the probabilistic mental model the domain requires, learned through deliberate practice. For that audience, evals are less a career pivot than an extension of what they already do well.

How to go deeper: the EVA lab

What this article covers is the map. Learning evals for real asks for something else: working a real case end to end, with the platforms the market is using today, making mistakes against your own outputs, and calibrating against human judgment.

That is exactly what our AI Evals program walks through. It runs across eight live sessions that take each participant from the mental-model shift the domain requires, through failure taxonomy, deterministic evals, LLM-as-judge, judge validation, RAG and agent evaluation, all the way to operating evals in production. It is built for two profiles that meet on the same ground: PMs who need quantitative judgment for shipping decisions, and testers moving into AI Quality, and it is designed so either can complete it without writing code. If your focus is the broader design and operation of agentic systems, the Certified Agentic AI Product Manager program treats evaluation as a core deliverable alongside design, guardrails, governance, and cost.

If the AI shipping decision makes you uncomfortable today, the discipline that closes that gap exists, it is maturing fast, and it can be learned. The question worth asking is not whether your team will need evals. It is when, and how expensive it will be to learn them live, in production, with real users watching.