Agent Evaluation and QA

An agent that was right in testing can drift quietly for months, and the first signal is usually a customer. The problem is that most teams never built a way to tell. We build the evaluation set from your own cases, score the agent against it on a schedule rather than on a hunch, and make a regression something that shows up before a release instead of after one.

Build the evaluation set, the scoring and the regression testing that tell you whether an AI agent is still right, before a customer does.

Agent evaluation and QA is the practice of building a fixed set of representative cases with known correct answers, scoring an AI agent against it, and re-running that scoring on every change and on a schedule, so that a drop in quality is detected by a test rather than by a user.

Agent evaluation means scoring an AI agent against a fixed set of your own cases with known right answers, on every change and on a schedule. Without it a drifting agent is invisible until someone complains. Thinklytics builds the evaluation set from real cases including the hard and rare ones, defines what counts as correct per case type, and wires the scoring into release so a regression blocks a deploy.

An evaluation set built from your real cases, including the awkward and rare ones, because an agent that only sees easy examples scores well and teaches you nothing.

A definition of correct that is specific per case type. For some tasks there is one right answer, for others a range, and conflating the two makes the score meaningless.

Scoring wired into release, so a regression blocks a deploy rather than being discovered after it.

Scheduled re-scoring, because the agent can stay the same while the model behind it, the data or the business rules change underneath it.

A benchmark score. How a model performs on a public test set says nothing about how it performs on your cases.

A one-off test before launch. The value is in running it repeatedly; a single pass tells you about one day.

A replacement for the review step. Evaluation tells you how often the agent is right in aggregate; the human in the loop catches the specific case in front of it.

A reason to keep tuning. Sometimes the honest finding is that the agent cannot reach the accuracy the task needs, and that is a result worth having early.

An evaluation set drawn from your real cases, with the distribution documented so you know what it does and does not cover.

Per case type, a written definition of what counts as a correct answer and what counts as an acceptable one.

Regression gating in the release path, so quality drops surface before deploy.

A drift report showing score over time against the same fixed set.

A documented process for adding new cases as reality produces them, so the set does not go stale.

Claims needing manual review, after scoring showed where the model was and was not reliable. $6.2M in suspicious claims flagged a year.

Blocked over a year on three separate member ID systems, with $4.8M a year in misrouted claims recovered once match accuracy was measured and lifted from 75 to 94 of 100.

No fixed evaluation set, so there is nothing to compare today against. Quality becomes a matter of opinion and recent anecdotes.

Scoring is not in the release path. A prompt change, a model version or a retrieval tweak can move quality and nothing blocks it.

The set was assembled from examples that were convenient to collect. The hard and rare cases are where the agent fails and where the cost sits.

Correct was never defined for that case type. Until it is, the score measures the grader rather than the agent.

Scoring an AI agent against a fixed set of cases where the right answer is known, so you can tell whether a change made it better or worse. It is the difference between knowing an agent is working and assuming it is because nobody has complained this week.

Smaller than people expect, and more carefully chosen than people expect. A few hundred well-chosen cases covering the common paths and the awkward edges beat several thousand convenient ones. What matters is that the distribution is documented, so you know what the score does not cover.

On every change to the prompt, the model, the retrieval or the rules, and on a schedule besides. The scheduled run is the one that catches drift, because the agent can be untouched while the model version, the source data or the business rules move underneath it.

It has to be defined per case type before scoring means anything. Some tasks have one right answer, some have a range of acceptable ones, and some are judged on whether the agent correctly declined to answer. Treating those the same produces a number nobody trusts.

No, and both are needed. Monitoring watches production traffic and tells you about volume, latency and errors. Evaluation scores a fixed set against known answers and tells you about quality. Monitoring cannot tell you the agent got quietly worse, because production has no answer key.

Then you have found that out cheaply and before a rollout, which is the point. Sometimes the fix is retrieval or prompting. Sometimes the honest answer is that the task needs a person, or needs the data fixed first, and that result is worth more than another month of tuning.

No. Evaluation tells you how often the agent is right across a set. Human review decides the specific case in front of it. A system with evaluation and no review has no safety net on the individual decision; one with review and no evaluation cannot tell whether it is drifting.

Three to five weeks for a single agent, and most of it goes on building the evaluation set and agreeing what correct means, not on the harness. Those two are the parts that need people who know the business, which is why they cannot be skipped.