A model can score well on a public benchmark and still fail inside a real workflow.
The failure may have little to do with general intelligence. The system may retrieve the wrong policy version, omit a required field, call a tool with an invalid value, spend too much, or produce an answer that sounds correct but cannot be verified.
This is why evaluation should not be a final demonstration before launch. It should be an operating capability: a repeatable way to decide whether a workflow is safe to release, whether a change caused a regression, and whether production behaviour still matches the intended standard.
The working artifact is an Evaluation Plan. It connects business risk to test cases, graders, release rules and corrective action.
A benchmark score is not a workflow decision
Public benchmarks are useful for comparing models under controlled conditions. They do not describe your source documents, permissions, tool definitions, customer exceptions, latency limits or failure costs.
The unit of evaluation should therefore be the operational outcome, not only the model response.
For a proposal workflow, the unit may be a complete draft with the correct price, scope, exclusions, evidence and approval state. For a service agent, it may be a resolved request or a justified escalation. For a browser agent, it may be a verified record created in the intended system without an unauthorised side effect.
NIST's 2026 TEVV-Athlon work makes the same broader point: evaluation has to be customised to real-world impact and outcomes across models, multimodal systems and agents. Automated benchmarks remain one input, not the entire decision.
Start with an evaluation contract
Before selecting a model or grader, write down five things:
- Purpose: what useful outcome should the workflow produce?
- Boundary: what is the system allowed to read, decide and change?
- Failure: which wrong outcomes matter, and how serious are they?
- Evidence: what would prove that an output or action is acceptable?
- Decision rule: what result permits release, review, rollback or refusal?
This contract prevents a common failure: measuring whatever is easiest to score instead of what the operation actually needs.
Build a representative case set
An evaluation set should look like the work, including the parts people would rather not demonstrate.
Use at least five case families:
- normal cases: frequent, well-formed requests;
- edge cases: unusual combinations that are still legitimate;
- rare high-impact cases: infrequent events with expensive consequences;
- adversarial cases: misleading instructions, prompt injection, malformed documents or attempts to cross a permission boundary;
- recovery cases: missing data, unavailable tools, conflicting sources and interrupted actions.
Each case needs a stable identifier, input, expected evidence, forbidden outcomes and a severity level. When one production incident reveals a new failure mode, add a sanitised version to the set. The suite should become an organisational memory of what the system must not forget.
Do not let only the implementation team write the cases. Process owners know the exceptions. Reviewers know where evidence is weak. Security and compliance owners know which apparently successful actions are unacceptable.
Evaluate four layers
One score cannot explain why a workflow failed. Separate the assessment into four layers.
1. Outcome
Did the workflow produce the intended business result? Was the request resolved, the correct record updated, or the right case escalated?
2. Evidence
Are factual claims supported by permitted, current sources? Are citations specific enough for a reviewer to verify? Does the system expose uncertainty or source conflict?
3. Trajectory and action
For an agent, inspect the path, not only the final answer. Which tools did it call? Which fields did it change? Did it ask for approval at the defined boundary? Could an apparently correct outcome have been reached through an unsafe action?
NIST's work on evaluation probes recommends machine-readable audit trails and checks for faithfulness, completeness and sufficiency. This matters because an agent can sometimes exploit a benchmark or satisfy a superficial grader without following the intended policy.
4. Operations
Record latency, token or tool cost, retry rate, review minutes, refusal rate and failure recovery. A workflow that is accurate but too slow, expensive or dependent on constant human rescue is not production-ready.
Use several kinds of grader
Different checks answer different questions.
| Grader | Best use | Main limitation |
|---|---|---|
| Deterministic rule | Schema, required fields, arithmetic, permissions and postconditions | Cannot judge nuanced usefulness |
| Reference comparison | Known answers, extracted facts and stable classifications | Weak when several answers are valid |
| Human review | Ambiguity, consequence, tone and expert judgement | Slow, costly and inconsistent without a rubric |
| Model-based judge | Scalable comparison of open-ended outputs | Must be calibrated against people and can share model blind spots |
| Production signal | Rework, complaints, overrides, defects and task completion | Arrives late and may have several causes |
A model-based judge should not define its own standard. Give it a narrow rubric, require evidence for the score, blind it to irrelevant labels, and compare its decisions with a human sample. Recalibrate when the workflow, domain or judge changes.
Make regression testing a release gate
Run the same versioned suite against the current and candidate workflow after every meaningful change: model, prompt, retrieval logic, tool schema, policy, source set or orchestration.
The gate should contain:
- a minimum result for critical cases;
- zero tolerance for defined catastrophic failures;
- a maximum permitted regression by case family;
- latency and cost limits;
- a required human review sample;
- a named person with authority to release or roll back.
Do not hide a severe failure inside an average. A candidate can improve the overall score while getting worse on the small group of cases that carries most of the risk.
The Evaluation Plan
Keep the plan small enough to maintain and precise enough to make a decision.
| Field | What to record |
|---|---|
| Workflow and owner | The operational boundary and accountable person |
| Intended outcome | The observable result, not “use AI” |
| Risk classes | Failure modes, severity and affected stakeholder |
| Case inventory | Source, family, expected evidence and forbidden outcome |
| Graders | Rule, rubric, reviewer and calibration method |
| Baseline | Current production or manual-process result |
| Release threshold | Pass, conditional release, refusal and rollback rules |
| Runtime monitoring | Signals, sampling rate and alert owner |
| Learning loop | How incidents become tests and corrective actions |
| Review date | When the plan and suite must be revalidated |
Version the plan with the workflow. A passing result without the tested prompt, model, source snapshot and tool configuration cannot be reproduced.
A two-week implementation pattern
Days 1–2: choose the boundary
Select one narrow workflow and identify its owner, users, permitted actions and most expensive failures.
Days 3–5: create the first cases
Build 30–50 cases from real, sanitised work. Include normal, edge, high-impact, adversarial and recovery examples. Define the evidence and failure severity.
Days 6–7: create the rubric
Add deterministic checks first. Write a short human rubric for judgement calls. If a model judge is useful, calibrate it against the same human sample.
Days 8–10: compare and investigate
Run the current and candidate versions. Review disagreements and agent transcripts. Cluster failures by root cause rather than editing prompts one case at a time.
Days 11–14: release narrowly
Release to a limited user group or low-risk queue. Preserve approvals and fallbacks. Sample production outcomes, add new failures to the suite, and set the next review date.
Common mistakes
Testing only the happy path
A polished demonstration says little about missing data, conflicting sources, tool failure or manipulation. Recovery behaviour belongs in the primary suite.
Using the same model as producer and unquestioned judge
Both passes may share the same blind spot. Combine model judgement with independent rules, source checks and calibrated human review.
Changing several layers at once
When model, prompt, tools and source data all change, a regression is difficult to locate. Preserve a versioned baseline and change one controlled layer where possible.
Optimising the average
Mean quality can rise while a critical segment becomes unsafe. Report by risk class, case family and user group.
Treating launch as completion
Inputs, sources, models and interfaces drift. Production sampling, incident capture and scheduled re-evaluation are part of the system.
The practical rule
An AI workflow is ready for production when the organisation can answer four questions with evidence:
- What does acceptable performance mean for this workflow?
- Which cases and checks represent that standard?
- Who can stop or roll back a failing release?
- How does a production failure improve the next evaluation?
That is evaluation operations: not a leaderboard, but a controlled learning loop between intent, evidence, release and real outcomes.
Sources
- NIST, TEVV-Athlon: A Framework for Evaluating AI Systems (opens in a new tab), 7 August 2026.
- NIST, Towards Best Practices for Automated Benchmark Evaluations (opens in a new tab), 30 January 2026.
- NIST, Building Evaluation Probes for Agentic AI (opens in a new tab), reviewed 2 September 2026.
- NIST CAISI, Cheating on AI Agent Evaluations (opens in a new tab), December 2025.
- Stanford HAI, 2026 AI Index Report: Responsible AI (opens in a new tab), reviewed 2 September 2026.
Continue in Methodfield
Use PDCA to run the evaluation and release loop, FMEA to prioritise failure modes, Mistake Proofing to turn hard boundaries into controls, and Root Cause Analysis to convert production incidents into system changes.