Diagram key: observation → shadow evaluation → recommendation → bounded execution → process participation → adaptive autonomy. Quality gates sit between stages, and the workflow can return to a safer mode.
AI automation projects often begin with the question: which actions can the system take over? A more reliable first question is:
Can AI first learn to recognise an acceptable result, its own uncertainty and the cases that must not be handled automatically?
At the beginning, AI can work as an observer. It receives a copy of a real case, evaluates it against defined criteria and records its conclusion without affecting a customer, payment, document or status. A person still makes the decision, while the team collects evidence about model quality on the real flow of work.
Authority expands only after proof: observation, shadow evaluation, recommendation, bounded action and finally variable autonomy inside predefined limits.
Why begin with control
Control reveals the real rules of the process
A procedure rarely contains every criterion used by an experienced employee. A specialist notices an unusual combination of factors, remembers the customer's history and knows when a formally correct result should still be rejected.
If AI is immediately allowed to act, these hidden criteria appear as operational errors. In shadow mode, a disagreement between the person and the system becomes evidence for investigation:
- a rule is missing or ambiguous;
- the model did not receive the required context;
- the specialist applies a criterion inconsistently;
- more than one reasonable decision exists;
- the model fails systematically on a particular type of input.
Observation therefore trains more than the AI. It helps the organisation describe its own work more accurately.
An observer's error does not become an action
A shadow system receives a copy of the input and produces an evaluation, but its answer is not returned to the user and does not change the working system. The team can see ordinary, rare and seasonal cases without creating an automated consequence.
This does not remove privacy obligations. A shadow deployment processes real data and requires its own access rights, retention period and usage log.
Trust becomes a measurable decision
A persuasive demonstration shows that a model can sometimes produce a good answer. It does not show where the model fails, whether it remains stable after an update or how it behaves with incomplete and unfamiliar inputs.
Parallel operation creates a reference set: the input, specialist decision, criteria applied, observed outcome and review of disputed cases. New model, prompt, knowledge-base and routing versions can be compared on this set before they are admitted to the operational workflow.
Checking the path and checking the result are different jobs
Conventional automation is well suited to checking sequence:
- are required fields complete;
- are the deadline, format and limit satisfied;
- was the necessary approval obtained;
- is the initiator authorised to act?
These are deterministic checks. They do not require a language model.
AI adds semantic evaluation of the result:
- does the document answer the original task;
- are claims supported by approved sources;
- are conclusions consistent with the input data;
- were the customer's constraints considered;
- do different sections contradict each other;
- is there enough information to decide?
In practical terms, this is not “human understanding”. It is an ability to reproduce expert evaluation for a bounded class of tasks with a measured level of agreement.
The companion article AI automation should improve quality explains why this additional layer matters to the business outcome.
The authority ladder
The stages below describe the operating mode of one specific workflow, not the overall maturity of a company. An organisation may use an autonomous action for low-risk classification while keeping AI as an observer in a financial process.
Stage 0. Define an acceptable outcome
Before connecting a model, the team records:
- what counts as a successful outcome;
- which requirements are mandatory;
- which errors are critical;
- what evidence must accompany a conclusion;
- who resolves disputed cases;
- which baseline measures will be used.
The output is a quality rubric with normal, boundary and unacceptable examples.
Stage 1. Observer
AI reads permitted data, extracts features and creates a log. It does not show a recommendation to the employee or change the workflow.
Evidence required: the data is accessible and sufficiently complete, privacy is respected and each case can be reconstructed for review.
Stage 2. Shadow evaluator
AI evaluates the result against the rubric in parallel. Work continues through the existing route; afterwards, both assessments are compared with the observed outcome.
Evidence required: quality is known for individual criteria, and recurring failure modes and blind spots are documented.
Stage 3. Assistant
The employee sees the criterion, suspected issue, supporting evidence and recommended action. They can accept, change or reject the recommendation.
Evidence required: the human–AI team produces a better result and does not automatically accept incorrect suggestions.
Stage 4. Bounded executor
The system performs only low-risk, reversible actions for known cases. Missing data, a new pattern or a critical rule violation leads to human review.
Evidence required: limits work, actions can be reversed and escalation keeps the relevant context.
Stage 5. Process participant
AI calls permitted tools and performs a short sequence of actions. Authority is limited by operation type, data, amount, customer and risk level.
Evidence required: the system remains observable after updates, the fallback route works and the process owner can stop execution.
Stage 6. Adaptive autonomy
The degree of independence changes with the situation. On a familiar, low-risk case, the system acts. On a new, ambiguous or sensitive case, it returns to recommendation or observation.
Mature autonomy is not maximal. It knows how to decrease.
Quality gates between stages
Authority should not increase according to a calendar or after a successful demo. Every transition needs its own evidence.
| Transition | What must be demonstrated |
|---|---|
| Observer → shadow evaluator | data completeness, correct access, stable processing and reliable logging |
| Shadow evaluator → assistant | quality by criterion, known failure modes and an acceptable rate of critical misses |
| Assistant → bounded executor | improved human–AI outcomes, appropriate employee response to wrong advice and explicit escalation rules |
| Bounded executor → process participant | reversible actions, tested limits, monitoring and a working fallback route |
| Process participant → adaptive autonomy | an ability to recognise the edge of competence and reduce authority automatically |
There is no universal threshold at which “95% means automate”. A false alarm and a missed violation have different costs. A marketing draft, a payment and a medical recommendation cannot use the same tolerance.
What real implementations illustrate
The following examples support individual patterns; they do not promise the same effect elsewhere. Most sources were published by the companies or technology providers and are not independent audits.
Morgan Stanley: criteria before scale
When developing its corporate assistant, specialists evaluated answers for accuracy and coherence before broad deployment. An OpenAI case study reports that access to relevant material increased from roughly 20% to 80%. Financial advisers can review and change the resulting meeting notes.
Pattern: the team formalised quality before scaling and retained specialist responsibility for the final material.
Moderna Dose ID: an independent check for a high-risk decision
The Dose ID pilot applied standard criteria to clinical data, checked dose selection and prepared a rationale with sources and charts. Experts performed a detailed final review; a separate quantified outcome for the pilot was not published.
Pattern: in a high-risk task, AI remains an additional evaluation layer rather than the owner of the decision.
CarMax: a draft passes through an editorial gate
AI produced summaries from customer reviews, while editors checked accuracy, context and brand voice before publication. In a Microsoft case study, the company reported that around 80% of draft summaries were approved by editors and that work estimated to require years manually was completed in months.
Pattern: a high acceptance rate did not remove independent approval of a public output.
Epilot: human evaluation before an operational action
The energy SaaS team first defined success criteria and compared models through human-based evaluations. The system then began preparing summaries and proposing actions that a user applies after review. An AWS case study reports an 87% time saving, but does not provide an independent audit or absolute baseline.
Pattern: evaluations were a gate before integration, not a report produced after launch.
Google: recommendations became bounded execution
Google's data-centre cooling system first recommended settings to operators and later applied them directly. Candidate actions were checked against constraints in both the cloud and the local control system, while operators retained control of the boundaries.
Pattern: autonomy grew after experience in recommendation mode and remained surrounded by independent checks.
A generator should not approve its own work unconditionally
If the same model creates an output and declares it correct without an independent mechanism, its blind spots may be shared. A persuasive explanation does not make a wrong conclusion true.
A reliable design combines:
- conventional code for exact constraints;
- a separate rubric for semantic evaluation;
- fact checking against approved sources;
- an independent second evaluation for consequential cases;
- human review of disagreements;
- measurement of the actual business outcome.
An AI evaluator must itself be evaluated. Research on LLM-as-a-judge shows that models can approach human ratings for some tasks, but results depend on the rubric, language, style and domain.
A small-business example
Consider a service company preparing commercial proposals.
The deterministic part checks company details, the current template, permitted discount and required attachments. AI can additionally evaluate whether:
- the proposal answers the customer's request;
- promises are supported by the service description;
- the timing is consistent with operational capacity;
- the next step is clear;
- a material constraint has been omitted.
A safe implementation sequence is:
- evaluate previously sent proposals without showing the result to the manager;
- review disagreements against the manager's decisions and sales outcomes;
- display findings before a proposal is sent;
- correct only formatting and required fields automatically;
- allow standard proposals to be prepared and routed;
- send unusual prices, promises and terms to the responsible employee.
What to measure
Average accuracy is not enough. A minimum measurement set includes:
- precision and recall for each criterion;
- false alarms and missed violations;
- quality on normal, rare and boundary cases;
- agreement among multiple experts;
- share of accepted, changed and rejected recommendations;
- human review time;
- observed outcome after the decision;
- frequency of incorrect autonomous actions;
- quality after a model or source update;
- share of cases where the system correctly identifies insufficient information.
Measure the interface separately. If an employee mechanically approves every suggestion, a nominal human-in-the-loop is not meaningful control.
Implementation plan
- Choose one frequent, verifiable workflow. Do not begin with an irreversible decision.
- Establish the baseline. Volume, manual time, waiting, errors, rework and exception cost.
- Write the outcome rubric. Separate exact, semantic and expert requirements.
- Run in shadow mode. Include normal cases, exceptions, incomplete data and integration failures.
- Review disagreements. Improve not only the prompt, but also rules, sources and the process itself.
- Show recommendations to a small group. Measure the human–AI team outcome.
- Permit one reversible action. Add a limit, audit trail and tested recovery path.
- Expand one authority at a time. Every new action passes its own gate.
Before choosing a level of autonomy, use Does the process need an AI agent?. For the permissions and recovery model, continue with AI agent authority. For the strategic context, read Flexible automation in a BANI world.
Final position
Beginning with control does not slow automation. It creates the evidence needed to accelerate it safely.
AI authority should grow only after the system has demonstrated an ability to:
- recognise an acceptable result;
- show the criteria and evidence used;
- detect missing information;
- escalate exceptions without losing context;
- remain inside verifiable limits;
- reduce autonomy when the situation changes.
The best first AI component is not the one immediately allowed to act. It is the one that helps the organisation see its own process and the boundary of safe automation more clearly.
Sources
- Amazon Web Services, Shadow tests — Amazon SageMaker AI (opens in a new tab), version reviewed 31 July 2026.
- Sana Tonekaboni et al., How to Validate Machine Learning Models Prior to Deployment (opens in a new tab), PMLR, 2022.
- NIST, AI Risk Management Framework Core (opens in a new tab), version reviewed 31 July 2026.
- Jinlan Fu et al., LLM-Rubric (opens in a new tab), ACL, 2024.
- Microsoft Research, Guidelines for Human-AI Interaction (opens in a new tab), 2019.
- OpenAI, Morgan Stanley customer story (opens in a new tab), version reviewed 31 July 2026.
- OpenAI, Moderna customer story (opens in a new tab), version reviewed 31 July 2026.
- Microsoft, CarMax customer story (opens in a new tab), version reviewed 31 July 2026.
- Amazon Web Services, Epilot generative AI case study (opens in a new tab), version reviewed 31 July 2026.
- Google DeepMind, Safety-first AI for autonomous data centre cooling and industrial control (opens in a new tab), 17 August 2018.