Sean Kelly, MD, chief medical officer at Imprivata, reflects on the necessary considerations around the deployment of agentic AI in healthcare.
Healthcare is under enormous strain. Clinicians, nurses, and staff are managing rising complexity, growing documentation demands, staffing shortages, and workflows that already feel stretched to the breaking point. When new technology promises to reduce burden and improve efficiency, it gets our attention quickly. This is part of what makes artificial intelligence (AI) particularly compelling in healthcare right now. We need help, and in many areas, AI is starting to offer it.
Go deeper with GlobalData
Access deeper industry intelligence
Experience unmatched clarity with a single platform that combines unique data, AI, and human expertise.
So far, much of the conversation has focused on assistive use cases: ambient documentation, summarisation, decision support, and other tools that help clinicians work more efficiently. Those applications matter, and many are already showing real value. But the next phase of AI in healthcare is different. We are moving from systems that support decisions to systems that can begin acting on them. That is where the stakes change.
Agentic AI
Agentic AI refers to systems that do not just identify tasks or recommend next steps, but can actually carry them out. These systems can initiate actions, move information across workflows, trigger communications, and in some cases complete operational tasks with limited human intervention. In the right settings, that could remove real friction from care delivery. It could reduce administrative burden, improve responsiveness, and help overextended teams function more effectively. I am optimistic about that future, but I also think we need to go into it with our eyes wide open.
In healthcare, there is no such thing as a purely administrative workflow. That is where a lot of risks get underestimated. On paper, it may seem reasonable to start agentic AI with “low-risk” operational use cases rather than overtly clinical ones. Scheduling is a good example: at first glance, it feels administrative, but in medicine, even scheduling decisions can carry serious clinical consequences.
Imagine a patient is referred for an ultrasound because of new leg swelling. An AI agent schedules that study for next week. On paper, the task is complete. Operationally, the system worked. But what if that patient’s swelling was actually caused by a deep vein thrombosis, and before the test occurs, the clot travels to the lungs and the patient ends up in the ICU with a pulmonary embolism?
That is not a hypothetical edge case. That is the kind of thing clinicians worry about because context matters. The urgency of that scheduling decision depends on details that may or may not be obvious in structured data alone. It depends on symptoms, co-morbidities, history, changes over time, and clinical judgment about what needs to happen now versus later.
Who owns the outcome?
So, if an agent made that scheduling decision, who owns the outcome? Is it the physician who placed the order? The staff member who would traditionally have scheduled it? The health system that approved the workflow? The vendor whose system enabled the action? The team that configured the agent? And, perhaps most importantly, did any human being actually see the decision at the point when intervention still mattered? The real issue with agentic AI in healthcare is not just that these systems can act, but that they can act inside workflows where small decisions may have hidden downstream consequences for patient safety.
The same concern applies to other use cases that appear straightforward until you look more closely. Take inbox management or medication refill workflows as examples. Most messages are routine. Most refill requests are appropriate. But hidden inside those high-volume tasks are the exceptions that matter most. A portal message that sounds minor may actually describe symptoms of a silent myocardial infarction in a patient with diabetes. A refill request may seem automatic until you realise the medication is no longer safe because the patient has not had their renal function checked. In those moments, the workflow is no longer administrative — it is clinical. If an autonomous system cannot reliably recognise that transition, it can create risks very quickly.
None of this is an argument against agentic AI. In fact, I believe the opposite. I think these technologies have enormous potential to help reduce burden on clinicians and staff, and healthcare urgently needs that kind of support. But we should be honest about where the risk lives. The challenge is not simply whether an AI agent is accurate in a narrow technical sense. The challenge is whether it can operate safely inside complex, interdependent care environments where timing, context, and exceptions matter as much as the task itself. That means governance cannot be treated as a policy document sitting on a shelf. It has to be operational.
Best practices for agentic AI management
In healthcare, we already understand how to work in high-risk environments. We do it every day. We rely on protocols, permissions, supervision, escalation paths, auditing, and defined scopes of practice. We do not give every human participant in the system unlimited authority, and we should not do that with AI agents either.
At the beginning, agentic systems should be tightly provisioned. They should be allowed to do only what they are specifically authorised to do, under clearly defined conditions, with human oversight built in. In many cases, there should be a human in the loop, not as a symbolic checkpoint, but as a meaningful part of the workflow. That is not a sign that the technology has failed. It is how healthcare manages responsibility and safety whenever the consequences are real.
Over time, some organisations will want to move beyond direct human review for every action. That may be appropriate for certain well-understood workflows. But if we are going to do that, we need the right controls in place. That starts with identity, access, monitoring, and auditability. If an AI agent is going to act inside a clinical or operational system, it should have a defined identity, a tightly constrained set of permissions, and clear rules around when it can act, what it can access, and how those actions are tracked. Behaviour should be observable, deviations should be detectable, actions should be attributable, and there should always be a way to pause, override, or revoke access when needed.
In other words, we need to govern AI agents more like participants in the care environment, not just software running in the background. This is where many organisations will need to shift their mindset. The question is not just, “What can this agent do?” The better question is, “Under what conditions should we trust it to act?”
Healthcare has always been cautious about technologies that insert themselves into clinical workflows, and for good reason. Friction is frustrating, but unsafe automation is worse. If agentic AI is going to succeed in healthcare, it will not be because it can move fast. It will be because it can move safely, predictably, and accountably within the realities of patient care. That is the standard. “First, do no harm.”
AI will absolutely play a larger role in healthcare. I believe it should. But when these systems begin acting rather than just advising, we have crossed into a different category of responsibility. At that point, we are no longer just evaluating a helpful tool. We are introducing a new operational actor into the system. And anything that can act in healthcare must be governed accordingly.
