“Which model is best value?” is incomplete until the job is defined. Extracting a few fields from regular invoices, drafting a sourced customer response and operating a multi-step agent demand different strengths. A public API rate helps estimate the lower bound of a bill. It cannot establish the quality, repair effort or risk of a result.
Consult current OpenAI (opens in a new tab), Anthropic (opens in a new tab), Google (opens in a new tab) and Mistral (opens in a new tab) API pricing when preparing a procurement comparison. A tariff is a procurement input, not a quality ranking. Different tokenizers, service tiers, context thresholds and tools make nominal rates a poor substitute for a measured workload.
Describe the work before selecting a candidate
Is the output closed and checkable, such as classification or field extraction? Or open-ended, such as analysis and design? What sources must be used? How many steps and tool calls are required? What happens if the answer is wrong? Who may approve an action? For an agent, specify whether it can read, propose, perform a reversible action or change an external system. These questions set the model requirements.
| Work | Starting class to test | Main failure to inspect |
|---|---|---|
| Repeated classification or extraction | Small, fast model | Rare formats, languages, omissions and refusal |
| Sourced document draft | Balanced model | Evidence, completeness and human edits |
| Difficult research or coding | Stronger reasoning model | Long-chain errors and time to completion |
| Tool-using agent | Model proven on that tool contract | Wrong tool, wrong arguments, endless loop, unsafe action |
These are test hypotheses, not claims that every model in a class performs alike. On a narrow task, a small model may outperform a larger one; on another, an expensive call can reduce retries and human repair.
Route work instead of forcing one model onto every case
A practical system may use a small model to classify a request, a search tool to locate evidence, a balanced model to prepare a standard answer and a stronger model for rare difficult cases. Escalation can follow missing evidence, conflicting sources, validator failure, high stakes or an action with external consequences. The router itself can be wrong, so audit its difficult cases and count its cost.
For every candidate, use the same real cases and required evidence. Allow each API its proper tool format, while holding the task and acceptance rule constant. Record actual billed tokens, all attempts, time to acceptance and human intervention. Set separate ceilings for severe errors. HELM (opens in a new tab) makes the underlying point: useful evaluation needs more than one scenario and one metric.
The 2026 AI Index (opens in a new tab) notes convergence among frontier models on some public rankings and problems with benchmark validity. A small difference in a general leaderboard is therefore a weak answer to a specific organization's document, tool and exception-handling problem.
Six requirements that change the shortlist
Routine-case quality. Does the candidate perform the main job, or does fluent prose conceal a missing field? Use a task rubric rather than an overall impression. For extraction, grade required fields and permissible omissions. For writing, grade mandatory facts and whether a reviewer can accept the draft.
Boundary-case quality. Test poor scans, conflicting documents, uncommon languages, incomplete policy and unknown formats. Here the real repair burden often becomes visible. Include cases where the correct result is “I do not know” or “approval is required.”
Tool and structure reliability. An agent must choose the correct tool, pass valid arguments, interpret errors, avoid unnecessary calls and stop when prohibited. Comparing agent models only on static questions misses central behavior. AgentBench (opens in a new tab) helped establish interactive evaluation across environments; its historical scores should not be treated as rankings of current models.
Data constraints. Check processing region, retention, auditability and contractual data rules. The lowest public price may not apply to the required regional or service configuration. This is a selection constraint from the beginning.
Time and predictability. A background batch job and an employee awaiting a response during a conversation need different processing modes. Look at time to an accepted result and tail latency, not just average tokens per second.
Stability and portability. Model versions, tokenizers and provider behavior can change. Keep a small regression set and a way to move work to another route. A business rule should not depend on one accidental phrasing of model output.
Make routing an explicit policy
The router does not have to be another large model. Begin with clear rules: document type, language, size, presence of required evidence and permission to act. Use a model-based classifier only if rules fail to cover the work. Define “difficult” by the job: an unfamiliar supplier for invoices, contradictory clauses for contracts, or an irreversible action for an agent. The escalation record should say why a stronger route or person was selected.
FrugalGPT (opens in a new tab) and RouteLLM (opens in a new tab) show what cascades and routing can achieve in studied task sets. Savings depend on identifying easy cases correctly. Sending a difficult case to a cheaper model can be far costlier than sending an easy case to a stronger one. Audit false negatives for risky cases, not only the average share of calls to the expensive model.
Example: an incoming-order agent
Decompose the work. Classifying the message can be checked against labeled history. Finding the current delivery policy can be checked against its source and version. Drafting a response can be graded for required facts and editorial acceptance. Updating an order is an external action, requiring permission, limits and an audit record. A small class may suit the first operation; a model strong at source use may suit the middle ones; the final operation needs a tested tool and bounded authority. One flagship model everywhere may add cost without improving outcomes. One cheap model everywhere may raise risk.
Set a separate threshold for each operation. If post-model review is expensive, a stronger candidate may be economical. If a decision is strictly deterministic, do not ask a model to “decide” a rule that can be expressed in code. It can interpret the incoming text and prepare an explanation; the rule itself can be executed reproducibly.
A comparison worth trusting
Build the sample before tuning the route. Include common cases, disputed exceptions, old documents, new formats, bilingual messages and requests for forbidden actions. Hold out a set that the prompt-design team does not inspect. Otherwise the configuration adapts to familiar examples and the measured quality becomes too optimistic.
Give candidates the same task and evidence and the same acceptance standard, while using each API's native tool and structured-output format. It is unfair to compare a model with working tools to one given only a textual description of them. Record model version, reasoning setting, length limits, test date and actual billed usage. Repeat important cases; one successful run does not establish reliability.
Use several gates. First are hard rules: no other customer's data, no unauthorized action, no invented order status. Second is quality: completeness, factual accuracy, evidence and clarity. Third is operation: elapsed time, full cost per accepted case, escalation rate and quota predictability. A candidate that fails the first gate cannot win because of a low tariff. This avoids a weighted score in which fast output appears to “compensate” for a dangerous error.
Three ways to supply models
One provider with several model classes simplifies integration and accounting, but concentrates dependence on its rates and service decisions. Several providers can offer resilience and task specialization, while complicating authentication, token accounting, monitoring and contractual review. Self-hosting an open model can control the environment and may pay off under a large steady workload, but requires engineering, hardware, updates, monitoring and quality measurement. Comparing only its marginal compute cost with an API token price ignores server utilization and on-call work.
Workload shape matters. Irregular requests are often easier to buy on demand. A stable high-volume flow may justify testing owned infrastructure or contracted capacity. Ownership of compute does not create quality by itself; the same local evaluation remains necessary. For a critical workflow, define graceful degradation. If the preferred model is unavailable or over budget, can a lower-risk case move to a fallback route, or should automatic action stop?
What to revisit after launch
Review the case mix as well as prices. A new language, channel, document type or longer customer history changes the work. A provider's lower price merits a new experiment; a higher public benchmark position only merits checking whether that model improves the local outcome. Migration cost includes prompt and tool changes, new tests and possible transition errors.
An economical model policy is documented and reversible. The team can explain why a case took a route, how to change that rule and how it will detect that exceptions are being missed.
A selection matrix without an artificial overall winner
After testing, create a short card for each operation: mandatory conditions, quality level, failure mode, cost per accepted result, latency, tools, data restrictions and fallback route. If a candidate fails a mandatory condition, do not award it a high overall score for price or speed. Among the viable choices, compare the differences that affect the job. For message sorting this may be missed rare categories; for contracts, a wrong clause reference; for agents, an unnecessary action.
Compare architectures as well as models. A strong model can perform poorly with stale evidence. A smaller model with good retrieval and a validator may produce a better accepted result. Test “model A plus full context,” “model A plus retrieval,” “model B plus retrieval,” and “route A to B under uncertainty.” Change one major component at a time so that the cause of improvement remains visible.
Agent tests should include behavior after failure. A tool times out, a status conflicts with another system, or a response is partial. Does the agent continue safely? Does it check whether an action already happened before retrying? Does it ask for human judgment when policy does not cover the case? These questions rarely appear in a model price table but determine production suitability.
Assign a model to a task route, not to the organization as a whole. Today a small class may answer ordinary delivery questions, a balanced class may prepare sourced document replies, a strong class may analyze disputed exceptions, and deterministic code may decide whether an order can be changed. The allocation can change in months. Good architecture lets the team revise roles without rebuilding the workflow.
Choose the least costly route that reliably clears the quality and safety threshold for the work.
Next: What AI efficiency means.