Mechanism
A system sounds convincing, so authority expands faster than evidence of safe performance.
Enabling conditions
- Evaluation rewards average agreement
- Reviewers work under time pressure
Failure modes
- Rare critical errors stay hidden
- Human review becomes automatic acceptance
Early signals
- • Explanations are trusted more than outcomes
- • Override rates fall toward zero
Countermeasures
- • Segment evaluations by consequence
- • Use staged autonomy
Diagnostic questions
- 1.Which decision competence was tested?
- 2.Where does the system fail outside the average?