Any doctor knows that the road to clinical licensure is long and winding. After medical school, it requires internship, residency, fellowships, written exams, and continuing education credits.
Internist and researcher Eric Bressman, MD, MHSP, assistant professor at the University of Pennsylvania Perelman School of Medicine, is deeply familiar with this process.
It’s why Bressman and his colleagues proposed, in a recent JAMA Internal Medicine article, creating a pathway similar to physician clinical licensure to help regulate artificial intelligence (AI) in medicine.
As with physicians, AI’s “road to licensure” would include rigorous training before release, as well as supervision while in use.
“These AI tools are being used actually pretty widely, but at the same time, they are sort of skirting any actual regulatory oversight,” says Bressman. “This doesn’t seem like a sustainable, long-term solution.”
AI “already has so much potential, both for benefit and harm. This piece really seemed to have a unique and new proposal for how to ensure the most effective, safest use of AI in medicine,” says Eve Rittenberg, MD, a primary care physician at Mass General Brigham, Harvard Medical School, Boston, who co-authored an accompanying editorial on Bressman’s proposal.
Bressman said the proposed framework is not a step-by-step guide for regulating AI but represents an alternative to current FDA approval pathways.
Keeping Pace With Tech Advancements
The FDA has historically regulated medical devices, from pacemakers to power wheelchairs, alongside prescription drugs. As computing costs decreased, a growing number of these devices began to include digital components.
Standalone apps and software designed to help interpret test results (such as ECG traces) or provide instructions during CPR resuscitation also began to crop up with increasing frequency.
So the FDA added a regulatory pathway known as Software as a Medical Device (SaMD). This pathway ranked software products by potential risk to patients and added validation and software update requirements.
AI-enabled devices, such as software to assist with reading radiology images or interpreting ECGs, had historically been pigeonholed into the SaMD pathway, Bressman said. And for those products, the strategy worked reasonably well. They were developed for specific uses and did not rely on answers created by generative AI.
Newer AI and large language models (LLMs), however, could have a broad range of uses. Some, like Open Evidence, generate answers for clinicians and create handouts tailored for specific patient concerns. Ambient AI scribes can take notes during patient encounters, facilitating billing and documentation.
Rittenberg, who uses multiple AI platforms in her own clinical practice, said the ambient scribe in particular has changed her life. It allows her to focus her attention on the patient and their care rather than her notes. With the AI absorbing some of the charting burden, she can leave her office on time almost every day, she said.
While Rittenberg’s experiences have been positive, newer AI tools for providers may pose greater risks for patients and providers. Feeding patient information to an AI agent or chatbot could potentially jeopardize privacy and lead to incorrect diagnoses.
Updates could subtly shift the AI’s responses away from the specific task for which it was approved. Biases embedded in algorithms could exacerbate existing inequalities. And not knowing how an AI is making a decision could create accountability concerns if something goes wrong.
“This is new. We’ve never had this issue before, and measuring it is complex,” says Majid Afshar, MD, a critical care and digital health physician in the Department of Medicine at the University of Wisconsin, Madison. “We’re still figuring out the governance and metrics to use.”
Of course, these types of errors happen all the time in human-only systems, which means that holding AI to some abstract standard of perfection may hamstring its usefulness, said Liam McCoy, MD MSc, a neurology resident and AI ethicist at the University of Alberta, Canada.
“Lower Risk” AI Products Regulated Differently
That’s why the 2016 21st Century Cures Act created exemptions for five different categories of software that were considered low risk and did not need to be regulated as a medical device.
One of those categories included clinical decision support software intended to help physicians be more certain about their diagnoses and therapy choices but not independently provide a diagnosis. The idea was to balance innovation with regulatory safeguards by exempting low-risk platforms from the most stringent oversight, Bressman says.
But nearly all AI platforms classify themselves as clinical decision support programs, which potentially leaves patients unprotected and at risk.
“We have a legislative process that takes time. It’s a long, deliberative process, and we have a technology that’s moving very quickly, in a health system that’s moving very quickly,” McCoy says.
To fill this gap, Bressman proposed a pathway not unlike clinical licensure for human physicians to be used in AI. Training the models is equivalent to medical school, and deploying the tools under close supervision would serve as internship and residency. This would allow LLMs and other AI models to continue to perfect skills before being turned loose upon patients. Passing certain exams and requiring ongoing clinical education to ensure the models don’t stray from their intended purposes round out the package.
“This is an ambitious proposal that will face many challenges and some resistance, and there are a lot of details to figure out,” Bressman says. “Perhaps the most important thing is having some more robust measure of oversight after you sort of let it out there, which is, I’d say, not the strength of the FDA.”
The FDA did not immediately respond to Medscape’s request for comment.
McCoy found the idea interesting but pointed out several caveats.
While AI tool can provide results that look like human responses, he said, their underlying programming means that they actually think nothing like a person. This makes it important to develop appropriate tests for AI rather than merely adapting existing board certification material for tests.
Evaluation of these models will also have to account for what McCoy calls their “fragile frontier” — how a model can perform exceedingly well at one test but fail abysmally at another.
Rather than expecting one regulation to be sufficient for all medical AI tools, McCoy says it’s likely that safeguards will accumulate piecemeal. Others will arrive due to malpractice lawsuits and other examples of tort law.
The development of such a strategy, in whatever form it takes, is essential, says Afshar.
“We have to come up with an acceptable framework that can balance but not hamper the speed of innovation, and also move us forward safely and cautiously,” Afshar says.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.