Two suppliers can demonstrate the same attractive assistant while hiding different operating costs and risks. One may use a private test dataset, another may have a human correcting outputs backstage, and neither may show what happens when an API fails. A fair comparison gives both suppliers the same work, the same permission boundary and the same acceptance rule.
Write the trial before the demo
Choose one workflow and define its observable result. Build a small set of sanitised cases from the business's normal work, difficult exceptions and recovery situations. For each case record expected evidence, permitted action, forbidden outcome and how completion will be verified. Ask both suppliers to run the cases under comparable conditions, including the same source data and integration scope.
Avoid a blended “quality score” that can hide a critical failure. Separate answer quality, action correctness, security boundary, reviewer effort, latency and recoverability. A supplier may pass the drafting task yet fail the permission test, which is a meaningful result rather than a low average.
Verify actions and access
Give the trial accounts the least privilege the workflow needs. Observe which data leaves each system, which subprocessors are involved, what is logged and how long records are retained. Ask the supplier to show the exact action sent to the business system and its returned transaction ID. A screen that says “done” is not confirmation that a CRM, calendar or payment system changed correctly.
Test a bad input, revoked permission, unavailable API and a partially completed run. The supplier should demonstrate a safe stop, status query, retry policy and a human takeover path. Review whether your team can understand the incident without supplier-only access.
Test the exit on day one
Export workflow definitions, records, prompts or configuration where relevant, evaluation cases and audit logs in a usable form. Import a sample or at least verify the schema and ownership of the files. An export button that produces an opaque archive is weak evidence of portability. Record what cannot leave the platform and the cost of replacing it.
The cost comparison should include subscription and usage, integration, review time, corrections, monitoring, support and exit effort. Use the same expected volume and case mix for both suppliers. The recent Methodfield AI economics series explains why token price alone cannot represent the cost of an accepted outcome.
A one-page decision record
For each supplier, record the same six fields: accepted cases, critical failures, human minutes, verified actions, export result and expected operating cost. Add unresolved questions, named owner and a decision date. Possible conclusions include buy, extend the trial for one uncertainty, reduce the proposed authority or stop. Do not treat an untested condition as passed.
This continues Methodfield's SaaS versus self-hosted framework. That article frames the architecture choice; this one specifies a comparative supplier trial. The case-evidence guide helps assess vendor claims, while a local trial checks fit to your own process.
Working artifact: the supplier scorecard
Prepare one scorecard and give the same copy to both suppliers before either demonstration.
| Field | Comparable evidence |
|---|---|
| Cases | Same inputs, source version, expected outcome and reviewer |
| Hard gates | Forbidden action, hidden access, missing log and failed recovery |
| Accepted work | Cases that meet the outcome and evidence rule after review |
| Portability | Readable export, ownership and sample re-import result |
| Cost | Fees, setup, human minutes, corrections and exit effort at one volume |
Report hard-gate failures separately from accepted-case counts. For cost, divide the same expected total operating cost by accepted outcomes and show the assumptions; if no case is accepted, the ratio is undefined and the trial has failed its outcome gate. A supplier that cannot demonstrate an action or an export has an unanswered requirement, not a pass with missing data.
Sources and scope
- Methodfield: Self-Hosted or SaaS?.
- Methodfield: What Does an AI Case Actually Prove?.
- Methodfield: The Full Cost of an AI Workflow.
Supplier prices, contract terms and data practices must be verified at the time of procurement.