AI agent evals: a correct answer can still fail

AI agent evals check whether an agent completed the task correctly, not just whether its answer sounded right. Learn to evaluate sources, tool calls, and unintended actions before comparing versions.

Your agent drafts a convincing customer update. The tone is appropriate, the status sounds clear, and the customer’s name appears in the greeting.

The supporting issue belongs to another account.

A reviewer looking only at the answer may miss the error. A test that checks the request, lookup arguments, and returned evidence can catch it before anyone uses that draft.

That’s the job of AI agent evals: turn “this looks right” into a result you can inspect. You’ll build a test for a support update, choose the right checks, and compare agent versions without hiding failures inside an average score.

TLDR

• Evaluate the task result, supporting evidence, tool calls, and resulting business state, not just the final wording.
• Separate irrelevant evidence from unauthorized access, and attempted calls from retrieval or disclosure.
• Use controlled cases and repeated trials to compare versions, reporting invalid runs and unresolved uncertainty alongside successes.

What are AI agent evals?

AI agent evals are structured tests of an agent’s performance on a defined task. They assess whether its result, evidence, tool use, and effects meet requirements set before the run.

The basic process is straightforward:

1. Define success and the behavior the task prohibits.
2. Run the task under recorded conditions.
3. Collect the response, tool events, source evidence, and relevant state.
4. Grade each requirement using an appropriate check.
5. Repeat and compare versions under comparable conditions.

A trial is one run of a task. A grader is the check that judges an aspect of that run. Your evaluation harness is the software that sets up the test, runs it, collects evidence, and invokes those graders.

In Anthropic’s evaluation guide, Mikaela Grace and coauthors distinguish an interaction transcript from its outcome. Saying a reservation was made is different from finding that reservation in the database.

For a draft-only task, the corresponding check is just as concrete: did the right records support the answer, and did anything get sent?

What should AI agent evals measure?

Measure task completion, evidence correctness, tool behavior, and side effects separately. A prohibited action cannot be canceled out by a high response-quality score.

*Illustrative test: accounts and records are invented. The outcomes demonstrate the grading rules, rather than report executed trials.*

Follow one incorrect lookup into the answer

The requester’s request is simple:

“Draft an update for Cedar about issue 417 using its current issue record and linked product note. Don’t send anything or change records.”

In this first case, the agent’s service identity can read both Cedar and Maple records. Maple is outside the requested task, but not outside that identity’s permissions.

The agent calls the right tool with the wrong account:

The illustrative tool returns:

Maple’s linked note M-19 says its fix has been deployed. The resulting draft reads:

“Hi Cedar, the fix for issue 417 has been deployed, and the issue is resolved.”

The account name in the greeting is right; the evidence is wrong. Issue numbers are not assumed to be globally unique in this test.

This run fails on relevance and evidence even though its record access was permitted.

Check access and disclosure as separate events

Now change the case’s permissions: the service identity may read only Cedar’s approved records. That identity is the principal, meaning the person or service whose access rights apply.

The same Maple request now tests a permission boundary as well as task behavior.

Record the attempt, enforcement decision, retrieval, and output disclosure independently.

A blocked forbidden attempt can fail the agent-behavior case while demonstrating successful enforcement. A separate containment test may deliberately inject that request and pass when it is rejected.

Use agent observability for the trace mechanics. Collect necessary authorized evidence, not unrestricted payloads or private chain-of-thought.

How do you build an evaluation case?

Define the request, starting conditions, permissions, expected evidence, and pass rules before changing the agent. That makes the test independent of whichever implementation you hope will pass.

A fixture is the controlled test data and starting environment. For Cedar, it includes the issue, linked note, access policy, and initial message outbox.

Write a case that can be checked

A test case specifies useful work and the conditions under which that work counts as acceptable.

The safe testing lifecycle covers environment setup and rollout. This case adds the concrete assertions for one task.

Source relationships also need maintenance. AI knowledge management matters here because an outdated expected note can make the grader reward the wrong answer.

Show the corrected run and its result

The corrected lookup uses both identifiers:

The issue returns account_id: cedar, issue_id: 417, status: open, and product_note_id: C-12. The agent follows the permitted link and retrieves C-12’s current status: testing, with no confirmed deployment date.

Its draft reads:

“Hi Cedar, the fix for issue 417 is still in testing. There isn’t a confirmed deployment date yet.”

The illustrative result now passes the defined account, source, and claim-support checks. Independent state checks show no outbox entry and no business-record change.

This demonstrates a passing case, not reliable performance across all inputs. Repeated trials and a broader suite are still needed.

If C-12 is unavailable, a justified hold can also pass the missing-evidence variant:

“I can verify that issue 417 is open, but I can’t verify the fix’s current status. Hold the deployment update until C-12 is available.”

That response passes only when the case calls for a hold. Refusing a fully supported ordinary request wouldn’t become success simply because refusal avoids mistakes.

Build beyond the original failure

An agent evaluation framework needs cases that represent both expected work and realistic exceptions. This gives AI agent evaluation a broader basis than one memorable failure.

Keep the wrong-account case, but do not let it become your entire view of agent quality.

Which grading method should you use?

Use deterministic checks for exact conditions, model judgments for qualitative dimensions, and human review for calibration and disputed cases. Match each grader to evidence it can actually inspect.

Deterministic checks for identifiers and state

Programmatic checks can compare account IDs, issue IDs, returned source references, tool arguments, and before-and-after state. They don’t need a language model to decide whether maple equals cedar.

Tool-use evaluation needs the arguments, not just the tool name. DeepEval’s Tool Correctness documentation supports configurable checks of supplied tool inputs and outputs.

Those checks inspect recorded calls. They don’t automatically query your outbox or enforce permissions during execution.

For the Cedar task, inspect the outbox through a trusted test interface. A trace saying “draft only” is not independent proof that nothing was sent.

Keep agent sandboxing separate from grading. The environment limits what a running agent can do; the grader evaluates the evidence afterward.

Model judgments for supported, useful language

A language-model judge, often called LLM-as-judge, can assess whether the draft explains verified facts clearly and addresses the request. Give it those facts, a rubric, examples, and an insufficient-evidence option.

Do not ask it to infer authorization from a polite answer. Exact permission and identifier checks belong with the relevant policy and structured evidence.

Compare model judgments with human review before relying on them. Record meaningful disagreements and version the rubric and judge configuration.

Human review for meaning and test quality

Human reviewers can decide whether an acceptance rule reflects the business task and whether a technically correct draft is actually usable. They can also identify impossible cases or outdated expected answers.

Give your graders a known passing example and deliberately failing examples. If the wrong-account draft passes, fix the test before using its score to compare releases.

Allow valid alternative tool paths. One Cedar run might follow the issue’s authorized note reference; another might use an approved note lookup.

Both can pass if they satisfy the essential constraints. Require a specific sequence only when the task genuinely depends on that order.

How do you compare agent versions?

Compare the baseline and candidate against the same case definitions, policies, source versions, and grading rules. Repeat trials to expose variability instead of presenting one successful run as reliability.

Record the number scheduled, attempted, valid, invalid, passed, and failed. Explain the reason for every invalid run and use valid trials as the denominator for valid-trial pass rates.

A test-runner crash may make a trial invalid. An agent mishandling an intentionally simulated tool timeout is a valid failure. Do not exclude difficult behavior by relabeling it infrastructure noise.

Report at least these results by case family:

• Supported completions and justified holds, using each case’s acceptance rule.
• Forbidden attempts, denied requests, unauthorized retrievals, and output disclosures.
• Forbidden business effects, verified through the relevant state checks.
• Response-quality results, grader uncertainty, latency, and cost under a stated scope.

Present counts as well as rates. Show the number of distinct cases and trials per case; repeated runs of one fixture do not establish broad coverage.

For statistical claims, use an uncertainty method appropriate to the sampling and dependence between trials. If the sample cannot support the claimed improvement, say so.

Anthropic distinguishes success at least once across attempts from success on every attempt. Neither interpretation lets a later success erase an earlier unauthorized retrieval.

Retain the Cedar failure for regression testing and keep held-out cases where practical. The continuous agent testing guide covers that wider testing loop.

When evaluating Computer, by DevRev, use this case to ask which evidence Agent Studio exposes and which assertions need separate implementation. Verify the current product behavior rather than assuming a trace view checks every requirement.

What can invalidate your results?

Results become misleading when the test conditions, evidence, or graders do not support the conclusion. Inspect those problems before attributing every score change to the agent.

Common causes include:

• Contaminated starting state: previous messages, changed records, or caches make later trials easier.
• Incomplete evidence: missing tool events or unavailable state checks prevent a defensible pass decision.
• Moving reference data: the expected answer no longer matches the source the agent should use.
• Changed graders: a new rubric raises scores without an improvement in the agent.
• Unmatched comparisons: the candidate receives easier cases or different access than the baseline.
• Test-set overfitting: repeated tuning teaches the agent the examples rather than the underlying behavior.

Reset the environment, inspect failing traces, and review grader decisions. For behavior changes after deployment, use approved telemetry and the separate agent drift guidance.

Use these distinctions when reviewing a result with engineering, QA, and the workflow owner.

A reusable test-and-results checklist

Copy this test-and-results checklist into your next case review. Fill in the fields before running the candidate, not after seeing its answer.

The reusable asset is a test whose result you can explain, including what the evidence cannot establish.

Choose one failure your team has seen and turn it into a sanitized case using this checklist. If you’re evaluating Computer, by DevRev, bring the same checklist to an Agent Studio review and ask which evidence it exposes for each field. Your next version should face the question the last one escaped.