Methodfield's AI Evaluation Operations recommends starting with a set of real, sanitised cases. A small business may have only a handful: the process is new, volumes are low, or sensitive cases cannot easily be shared. That does not prevent testing. It changes what the test can honestly establish. A small set can reveal failure modes; it cannot prove a broad success rate.
Start with consequence, not a percentage
Write the workflow boundary and the worst plausible wrong action. Is the system drafting a low-risk answer, changing a customer record or sending a financial instruction? Name who can detect and reverse an error. Separate failures that block release from defects that can be corrected in a limited trial.
Collect all usable real cases with permission and remove unnecessary personal data. Keep their provenance: when they occurred, which channel, what outcome the employee reached and what evidence supported it. Do not manufacture a large dataset by slightly rewording the same five examples and then treat it as independent evidence.
Add targeted probes
Real history may contain no rare but consequential events. Build deliberate probes for missing fields, conflicting records, a revoked permission, tool timeout, malicious instructions inside a document and a customer request outside the system's authority. Mark these as designed tests, not representative production frequency. They answer a different question: can the workflow stop safely when the situation occurs?
Use independent checks where possible. A rule can verify that a returned ID exists, a source link points to the approved document, a date falls within policy and a write affects the intended record. A qualified person judges ambiguous language, fairness and whether the response is actually useful. Record the reasons for disagreement rather than editing the prompt after each single miss.
Compare the baseline case by case
For each case, show the current human or ordinary-software result beside the AI-assisted result. Record accepted, edited, rejected and unsafe outcomes, plus reviewer minutes. With a small set, show counts and the cases themselves. “Nine of ten passed” may describe those ten examples; it does not justify a claim that the system is 90% accurate in future work.
Suppose an equipment-rental firm has twelve sanitised booking-change requests. Ten are routine, one has conflicting dates and one concerns the last available machine already promised to another customer. The trial can reveal a conflict or a change to the wrong booking. It cannot establish a stable performance rate for all future rental requests. The example illustrates the inference boundary, not an observed result.
Make launch earn more evidence
If critical probes pass and the operational owner accepts the remaining uncertainty, release only to a low-risk queue with human approval. Save each new case, correction and incident for the next test set. Set a volume or time trigger for re-evaluation. If the business cannot detect a harmful outcome, cannot identify an owner or cannot recover a bad action, the safer decision is to keep the system in draft or shadow mode.
This article extends Methodfield's general Evaluation Plan for a common SME constraint: evidence scarcity. It also uses the case-evidence ladder to keep observed results separate from claims about wider performance.
Working artifact: a small-case evidence register
The register protects the distinction between observed production work and deliberately designed stress tests.
| Field | What the reviewer records |
|---|---|
| Origin | Real authorised case, scripted probe or later production sample |
| Expected | Acceptable outcome, required evidence and forbidden action |
| Observed | Actual output, tool path, changed record and reviewer edits |
| Severity | Harmless defect, recoverable error or release-blocking failure |
| Next step | Fix, owner, retest date and permitted launch scope |
Keep the full denominator visible: how many distinct real cases, how many probes and how many repeated variants. A pass on a designed probe shows that one failure path was handled in the test; it does not estimate how often that path occurs in the business.
Sources and scope
The suggested test is an operating method, not a statistical guarantee or a substitute for domain-specific validation.