AI governance risks human judgment; high-reliability lessons offer solution
New frameworks may turn proactive threat detection into reactive alert fatigue by undermining front-line authority.
A hidden problem is emerging in artificial intelligence governance across high-risk industries: by making humans react to AI-generated alerts instead of proactively identifying potential threats, these frameworks are subtly eroding critical human behaviors. Instead of reinforcing safety culture, current approaches risk replacing deep operational judgment with a cycle of reactive alert management that leaves organizations vulnerable when systems fail or produce noise.
The High-Reliability Blueprint
Operations running under extreme risk — such as nuclear power plants, aircraft carriers, and air traffic control centers — have long understood that safety depends not just on good technology but on how people exercise judgment under uncertainty. The scholarly field of high-reliability organizing (HRO) has identified a clear normative framework to distinguish organizations that manage catastrophic risk well from those that succumb to it:
- Reliable organizations protect front-line judgment
- Distribute decision authority to those closest to the problem
- Treat emerging anomalies as signals rather than noise
These principles are hard-won lessons from disasters. At Three Mile Island in 1979, operators of the U.S. nuclear power plant confronted more than 100 simultaneous alarms with no prioritization mechanism, producing cognitive paralysis rather than an informed response when the reactor began to melt down. Before the space shuttle Columbia’s deadly final flight in 2003, NASA engineers who suspected that a wing’s insulating foam had sustained damage during launch had escalation channels available but refrained from using them due to the organizational costs of challenging senior leaders. NASA’s accident investigation report said that “management techniques unknowingly imposed barriers that kept at bay both engineering concerns and dissenting views, and ultimately helped create ‘blind spots’ that prevented them from seeing the danger the foam strike posed.”
The AI Governance Paradox
Today’s AI-enabled threat-detection systems — the adaptive machine-learning tools that surface fraudulent transactions, network intrusions, compliance violations, early signs of patient deterioration in nursing care, and the like — promise superhuman pattern recognition at scale. These threat-detection technologies, however, remain prone to bias, drift, and hallucination, and pose their own risks. A false positive can flood an organization with noise and lead to wasted resources. A missed threat can be catastrophic, as the cyber hacks a few years ago by Salt Typhoon against U.S. telecom networks and the massive 2024 data breach of Change Healthcare have shown.
Emerging AI governance frameworks, designed in part to protect organizations from AI technology falling short of its intended task, are paralleling underlying principles of the original HRO playbook (whether they realize it or not).
