The market for artificial intelligence (AI)-based devices in the medical imaging space continues to grow rapidly, with no indication that demand or innovation will slow in the near term. With this ongoing rise, governance strategies to manage the safety, data accuracy and performance of AI devices and software, both at the developmental stage and throughout a product’s lifecycle, are of increasing importance.
AI is having a significant impact across healthcare, not least in the medical imaging space. A report by GlobalData forecasts that AI in healthcare will reach a valuation of $57.4bn in 2029. Looking at the FDA’s list of AI-based medical devices approved in 2026, the overwhelming majority are products designed for applications in the medical imaging space.
Go deeper with GlobalData
Discover B2B Marketing That Performs
Combine business intelligence and editorial excellence to reach engaged professionals across 36 leading media platforms.
In Europe, regulation surrounding AI-based medical devices and software is advancing in the form of the EU AI Act, initially implemented in 2024. Similarly, the UK’s Medicines and Healthcare products Regulatory Agency (MHRA) is currently overhauling its regulatory approach to AI in healthcare to account for gaps in existing regulation.
AI in the US is currently regulated primarily through a state-by-state patchwork of legal obligations, with the scope and structure of potential federal regulation still uncertain.
Despite an executive order (EO) issued by President Trump in January 2025 that pledged to leave AI innovators “free to innovate without cumbersome regulation”, the US Food and Drug Administration (FDA) has recently issued several draft guidance documents regarding AI device and software evaluation, including recommendations for predetermined change control plans (PCCPs), with a sharp focus on a product’s full lifecycle. The FDA also entered a more formal collaboration with the MHRA in June that will focus on shaping global healthcare regulation in areas including AI-based medical devices.
Upcoming events such as the US mid-terms in November could shift the nation’s regulatory approach at the federal level. And while many medtech companies in the US with AI products are frustrated by the uncertain nature of AI case law, the perspective of market observers is that it is prudent to prepare for stricter regulatory edicts in the future.
Focus on AI governance, regardless of current regulation
In guiding medical device companies on governing their AI-based products, A.J. Bahou, partner at law firm Bradley Arant Boult Cummings, tells Medical Device Network that he advises clients to “build governance in” to their AI products, regardless of current regulatory schemes as AI case law in the US continues to evolve.
Bahou highlights that in the US, the stance around AI is an ongoing push for tighter regulation at the state level, yet at the federal level, there is currently more of an emphasis on innovation.
“The struggle for everyone in the industry comes down to uncertainty regarding federal versus state pre-emption”, he says
To mediate this issue, when clients ask Bahou about dealing with the shifting nature of AI regulation, he refocuses their attention on the question of ‘what can you do to have a safe and secure device?’
In this regard, tenets that need building into a device include security and related protocols that account for the risk potential of a device.
“If you build those factors in, that is the governance structure for the full lifecycle of the device, then you tend to relax the conversation and stop fighting over what Colorado has versus what California has versus federal pre-emption,” Bahou explains.
Mirroring these points, Patrick Mans, head of data and AI engineering, innovation and design at Philips, explains that the Netherlands-headquartered company’s approach is to embed AI governance principles into its products “from the beginning”.
“Rather than treating compliance as a series of separate regulatory exercises, we build AI on robust quality systems and governance processes that support responsible innovation and help us meet applicable requirements across global markets,” Mans explains.
AI governance throughout the device lifecycle
For AI devices and software underpinned by models, these are subject to change over time, meaning the onus is now on manufacturers to ensure their AI is functioning properly throughout a device’s entire lifecycle.
“Companies should be hyper-focused on the developmental factors for their AI devices, but they should also consider – once their device is FDA-approved and launched in the marketplace – what will happen in, say, six months when their device’s AI model gets updated, and to bear in mind what impact those changes could have on real patients,” says Bahou.
“As such, manufacturers need to continually test and monitor their devices throughout their lifecycle post-market.”
This shift represents a stark contrast to past operating procedures, Bahou adds, as once a company got a medical device approved and “its software locked down”, they were previously “very resistant to changing anything about their device”.
Mans notes that AI governance starts “long before a product reaches the market”. During development, Phillips applies “established quality and risk management processes” and validates AI performance using representative datasets for its intended clinical use, he says, and also assesses factors including transparency, human oversight, cybersecurity and privacy.
“We also consider how AI integrates into clinical workflows, ensuring users understand both the capabilities and limitations of the technology,” Mans adds.
Mans emphasises that post-market surveillance must also be viewed as an essential part of AI governance.
“Healthcare environments, clinical practice and technology continue to evolve, making it important to monitor how AI performs in real-world use,” he says.
“Philips applies established post-market quality processes to AI-enabled products, including monitoring field performance, evaluating customer feedback, managing cybersecurity, and investigating potential issues where appropriate.”
Regarding AI, these stipulations also include monitoring for changes in real-world performance so any emerging issues can be “assessed through established quality processes”, Mans adds.
Designates of a successful AI governance strategy
As AI becomes a key part of medical imaging devices such as MRI scanners and X-ray machines, companies must be aware that the technology introduces a variety of new risks that they need to be able to mitigate through effective governance protocols.
Erez Kaminski, CEO of software development and regulatory compliance company Ketryx, believes that many companies view AI governance as a process problem.
“But for a lot of companies, I think it’s more of an infrastructure problem, in that they lack the infrastructure to build AI and govern it effectively,” he says.
A well-rationalised AI governance strategy is predicated on companies’ understanding where their AI model’s data is coming from, how it can be curated, and the procedures that are in place to properly apply quality assurance protocols to it.
“The biggest point then relates to how a company can iteratively validate and manufacture their software, particularly if they are using very new AI models that may need to be updated quite frequently,” Kaminski says.
“Even updating once a quarter is something that most companies don’t know how to do or have real challenges in doing,” he adds.
Kaminski likens this challenge to companies’ lacking a sufficient “AI assembly line”, giving rise to the question of whether a company is governing their AI properly.
For effective AI governance, a company’s ability to capture and apply it to a given product is “table stakes”, Kaminski notes. However, other stipulations include having AI systems in the right infrastructural environment to train them effectively, and ensuring the data makes sense so that it is not “so far removed from the intended purpose of a device or patient population that it corrupts a company’s model”.
Meanwhile, further considerations should centre around a company’s quality assurance and traceability provisions.
“Companies need to know how they are going to trace all of the different aspects that ensure their AI is effective, safe, compliant, and providing good data, and that they have proper tests and reviews in place to monitor these factors over time,” Kaminski explains.
“And finally, companies need to ensure that their change management protocols are fast. AI products have change management on the order of weeks and months, yet many companies operate on the order of years.”
The inherent challenges of AI governance equally apply to the healthcare institutions that adopted AI-based medical imaging devices into their workflows.
In setting up an effective governance plan, Bahou advises clients to establish a governance committee comprised of many different stakeholders.
“Such committees should not only be comprised of IT personnel,” Bahou stresses. “They should also include persons such as the chief medical officer, chief nursing officer, and people from an institution’s finance, cybersecurity, and data privacy teams.”
Via audits and device monitoring, a diversified AI governance team holds the ability to more quickly recognise if an AI tool’s performance is deviating, potentially skewing results or exacerbating biases due to a situation commonly referred to as AI drift.
Caused by factors such as AI model updates and shifts in the relationships between input and output variables, AI drift can land device or software manufacturers in legal trouble as it can cause their device to deviate from its regulatory-approved function, while also holding the potential for misdiagnosis, thereby causing patient harm.
AI governance is a broad-reaching process that requires careful consideration, both for device manufacturers at the developmental and post-market stages, and for adopters of the technology. With the right governance strategy in place, developers have a means to deal with the challenges AI brings with it. Meanwhile, for the technology’s adopters, governance and oversight are critical in ensuring that a given AI device is driving efficiencies and performing its core function as expected and not giving rise to new problems and potential legal exposures around diagnostic protocols for patients.
