Healthcare organizations are rapidly integrating artificial intelligence into clinical decision-making, often assuming that strong model performance ensures system safety.
That assumption is flawed.
Safety in AI-enabled healthcare depends not on the model itself, but on the workflow in which it operates. This is an operational, not just theoretical, distinction that determines where risk exists.
The Illusion of Safe Automation
In 2013, Asiana Airlines Flight 214 approached San Francisco on what should have been a routine landing. The weather was clear, the aircraft was functioning normally, and the crew was experienced. Nothing about the situation suggested that the flight would end in disaster.
As the aircraft descended, the pilots managed the approach with automated systems. Like modern clinical environments, the cockpit was neither just a machine nor just a person in control, but a system in which pilots and technology worked together, each influencing the other.
During descent, the crew selected an automation mode for speed and altitude, but it behaved unexpectedly. The autothrottle, usually responsible for maintaining airspeed, ceased to do so. This subtle shift required pilot recognition and interpretation, but it went unnoticed.
The aircraft continued descending, with the automation performing exactly as programmed. The pilots, believing the system was still protecting the critical airspeed, acted accordingly. With thrust remaining at idle, airspeed began to decay.
At first, nothing seemed wrong. System failures often unfold gradually, not as a single event, but as a widening gap between the system’s actions and the operators’ perceptions. The pilots managed the aircraft, but their understanding of how the system controlled the descent was incorrect.
As the approach continued, pilot workload increased. The aircraft drifted off course, and visual, instrument, and glidepath cues all signaled increasing instability. The crew had to interpret, prioritize, and act within a compressed timeframe.
At 500 feet, the situation reached a threshold. By standard procedure, an unstable approach required a go-around. This was not just a judgment call. It was a structured control point, a time when the human-technology team was expected to act.
That response did not occur.
The aircraft continued to descend as the airspeed dropped. The crew tried to recover only at the last second, but it was too late; the plane struck the seawall short of the runway.
The systems had not failed. Nor had the pilots simply “made a mistake.” The failure arose from their interaction: how the crew understood the system, how the system behaved, and how the governing workflow performed under stress.
The accident reflected a convergence of factors: complex system behavior, an incomplete understanding of automation modes, increased workload, and reliance on technology requiring active monitoring. Most importantly, it reflected a failure to act at a defined control point within a workflow that depends on both human judgment and technological behavior functioning together.
Aviation did not respond by abandoning automation or by placing a greater burden solely on pilots. Instead, it strengthened the whole system: training methods, technology behaviors, and workflow rules for human intervention. The goal was not to eliminate human involvement or technological complexity, but to ensure their interaction is structured, observable, and controlled.
Healthcare is now introducing AI into clinical workflows under similarly complex conditions, often without the same level of system discipline.
Human Oversight Is Necessary—but Not Sufficient
A common response to these risks is to ensure that a human remains involved in decisions influenced by AI. This is necessary, but it is not sufficient.
Human judgment is crucial in clinical care. But it is not a reliable safety control if it’s the main way to manage AI risk. Clinicians work under time pressure, mental load, and many demands. These conditions limit their ability to spot and fix errors in AI recommendations.
If safety depends on a clinician’s ability to recognize when an AI recommendation is incorrect, the workflow is inherently fragile.
Healthcare has encountered this pattern before. Clinical decision support alerts are often overridden. Recommendations are accepted without sufficient scrutiny or ignored entirely. When AI is embedded into workflows that prioritize efficiency, it can shape behavior in unintended ways. More advanced AI will not eliminate these dynamics. It will intensify them.
The implication is clear. A human “in the loop” is not, by itself, a safety strategy.
A human “stop” is not a person. It is a designated control point within the workflow that governs how AI is used.
Safety requires structuring workflows so AI recommendations cannot affect care without deliberate engagement at defined points. These control points determine how recommendations are evaluated, validated, and challenged.
This approach does not limit clinical autonomy. It lets clinicians exercise judgment within a workflow that reduces AI errors and addresses known human limitations. Without this, organizations lack real decision support. They let AI shape clinical decisions without enough control.
Therefore, the heart of the challenge is not the AI technology itself, but how workflows are designed to ensure safety when using AI.
Bringing AI into clinics means understanding practical decision-making and how AI changes it. Organizations must understand where AI helps, where it poses risks, and how it affects clinicians in real-world conditions. Poor integration means AI increases mental load, adds confusion, and opens new ways to fail.
Well-designed workflows do the opposite. They make things clear, guide AI use, and create patterns that can be measured and improved.
This leads to a critical conclusion. There is no static model for safe AI deployment.
Delivering Quality Outcomes
AI operates within dynamic clinical environments. Its impact emerges over time. Safe implementation requires ongoing observation, measurement, and refinement, not just of the AI, but also its use in clinical workflows.
Aviation got safer through systematic reporting and shared learning. Healthcare has no system like this for AI. AI tools are used in isolation, failures are inconsistently reported, and groups rarely share what they learn. This leaves each organization managing AI risk on its own.
This fragmentation is not sustainable. If AI is to be safely integrated into clinical care, healthcare must move toward a model of shared learning, where real-world use informs continuous improvement.
The stakes extend beyond performance. Healthcare is built on trust, between patients and clinicians, and between clinicians and the technologies that support their decisions. Poorly integrated AI undermines that trust.
Leaders must confront this now. The question is not if an AI model works on its own. The real question is whether AI, as used in workflows, leads to safe, reliable results.
If leaders cannot identify where AI influences decisions, where control points exist, or how performance is monitored and refined, they accept risks they do not fully understand.
Conclusion
Artificial intelligence will become an integral part of healthcare delivery. That outcome is not in question. What remains uncertain is whether its integration will consistently improve care or introduce new forms of harm.
The difference will not be determined by the sophistication of AI models. It will be determined by the discipline with which healthcare organizations design and manage the workflows in which those models are used.
The lesson is clear. Human judgment is necessary, but it is not a sufficient safety control. Simply placing clinicians “in the loop” does not ensure safe outcomes when workflows allow decisions to become automatic, unexamined, or poorly understood.
Safety must be intentionally designed into how AI is used. It must be reflected in clearly defined control points, structured interactions, and continuous observation of how decisions are made in practice. This is not a one-time implementation. It is an ongoing responsibility.
Healthcare has faced similar moments before. Other industries have as well. Those that achieved high levels of safety did so not by limiting technology, but by strengthening the systems in which technology operates.
Healthcare now faces the same challenge.
If organizations continue to focus on model performance while neglecting workflow design, they will introduce risk they do not fully understand. If they instead commit to designing, monitoring, and continuously improving the integration of AI into clinical care, they can realize its benefits while protecting patients and preserving trust.
AI will not determine the future of healthcare.
The way it is used will.
