HHS wants an AI agent that can act as a round-the-clock member of a heart failure patient’s care team, managing care between visits and escalating to a clinician only when needed.
On Sept. 9, the department’s Advanced Research Projects Agency for Health selected six organizations to build it under a four-year, $62.7 million contract called the Agentic AI-EnableD CardioVascular CAre TransfOrmation program, or ADVOCATE.
Three companies — Atman Health, Tempus AI and Updoc — will build the patient-facing AI agents. Stanford (Calif.) University will build a separate supervisory AI system to monitor those agents for unsafe recommendations. Durham, N.C.-based Duke University and Oakland, Calif.-based Kaiser Permanente will lead real-world deployment across dozens of hospitals and clinics, with the lead developer required to submit an FDA authorization package within 24 months.
“These are the expected and logical developments in this space,” Curtis Cole, MD, chief global information officer of Ithaca, N.Y.-based Cornell University, told Becker’s. “The creation of agents that can make autonomous decisions in a limited domain with oversight by a system created by a third party seems like the right approach technically. As they master the edge cases, they can incrementally expand the scope.”
The design, he said, is not where the risk sits.
“My primary issue remains — who will carry the liability? If the companies that build and oversee the agents have no liability, I don’t see how these will succeed — unless we eliminate the whole concept of liability, like has been done with gun manufacturers. That seems less likely in healthcare where Americans have such a long history of holding someone accountable,” Dr. Cole said.
Darrell Bodnar, CIO of Whitefield, N.H.-based North Country Healthcare, drew a sharper line between AI that extends what clinicians can do and AI that acts without them.
“I am encouraged by the concept of clinicians, under federal funding and within a controlled pilot environment, pushing the boundaries of how AI can safely extend care and improve access,” he said. “Where I become much more concerned is the movement toward autonomous AI making clinical decisions, including but not limited to adjustments to medications, diet, and treatment, without direct human oversight.”
Mr. Bodnar said that oversight has to be built into the model, not layered on after the fact.
“That oversight needs to be grounded in rigorous, evidence-based clinical science, with clear accountability for the decisions being made. Accountability for a clinical decision needs to be clearly defined and ultimately owned by someone,” he said.
He added that his confidence in federal regulators to independently guarantee that oversight “has been somewhat diminished by some of the changes and decisions we have seen recently” at the FDA.
“I do support FDA approval as one safeguard — it should not replace transparent, evidence-based clinical governance and meaningful human oversight,” he said.
For Joy Oh, chief digital information officer at Cincinnati-based Christ Hospital Health Network, the most interesting part of ADVOCATE is not the clinician-facing agent — it’s the second one. Stanford’s supervisory system, tasked with monitoring the clinical agents and stopping any recommendation that falls outside protocol, is the piece she said could unlock faster, safer deployment of agentic AI more broadly.
“If this framework proves successful, I believe it could open the doors for more rapid prototyping, testing, and deployment of agentic AI at scale by reducing reliance on continuous human oversight and training,” she said.
But a second layer of AI raises a second layer of the same question.
“If a supervisory agent fails to detect an inappropriate recommendation that results in patient harm, who is ultimately responsible: the developer of the worker agent or the developer of the supervisory agent?” Ms. Oh said. “Could the presence of a supervisory agent create a sense of false security leading to long stretches of time where drift and hallucinations go unchecked and undiscovered?”
Brett Oliver, MD, chief medical information and AI officer at Louisville, Ky.-based Baptist Health, sees the funding decision itself as a message.
“I think this is one of the more important AI initiatives in U.S. healthcare today, not because of the technology itself, but because it signals an important shift in regulatory thinking,” Dr. Oliver said. “This investment appears to mark a move from using AI as decision support to AI that actually performs a portion of care itself. Human-in-the-loop may no longer be the primary safety design model (see also Utah’s Doctronic medication refill initiative), but rather a transitional governance model. The future is likely to involve exception-based oversight.”
He said it’s no longer a matter of whether autonomous agents become part of clinical care.
“The question is one of timing,” Dr. Oliver said. “Each healthcare system needs to ask whether they are ready with an appropriate governance structure for these tools. Can we monitor drift? Can we intervene when autonomous systems behave unexpectedly?”
Ashis Barad, MD, chief transformation and digital information officer at New York City-based Hospital for Special Surgery, said the ADVOCATE award does not change his underlying philosophy on the technology.
“The responsible path for autonomous clinical AI is to begin with narrow, well-defined use cases, with clear clinical accountability, rigorous performance monitoring, predefined escalation pathways, and easy access to a human,” Dr. Barad said. “Broad autonomy across complex care remains a much higher bar.”
He argued the real opportunity is not autonomy for its own sake, but coverage — the space between visits where patients are currently on their own.
“The breakthrough is not autonomous AI. It is giving every patient continuous support when human care teams cannot be everywhere at once,” he said.
Daniel Kortsch, MD, associate chief of AI and digital health at Denver Health, raised a different accountability question entirely: to whom is this technology accountable, and on whom was it actually tested? Denver Health is the city’s safety-net system; Dr. Kortsch said many of his patients are uninsured or on Medicaid, several speak a language other than English, and some share a single phone with spotty data across a household.
“When I read about a program like this, I’m reading it for the patient who has heart failure, no cardiologist, and a phone that is shared between multiple people and doesn’t always have data,” Dr. Kortsch said. “Autonomous clinical AI is coming — but the question is for whom.”
“The evidence for AI expertise is real,” he added. “The JAMA Viewpoint last month showed models matching or beating physicians on history, diagnosis, test selection, and guideline-based prescribing. But those are curated cases without real-world data.”
Dr. Kortsch also noted that the piece’s co-authors, investors Vinod and Neal Khosla, back ADVOCATE awardee Atman Health, while Neal “has a patent pending on the kind of AI-supervises-AI architecture ARPA-H is now funding.”
“Unfortunately, guideline-concordant therapy is the easy part,” Dr. Kortsch said. “Titrating a heart failure drug in someone who fills half their prescriptions because of cost is the challenge. The JAMA authors count chronic disease as something AI already does better, but their evidence is four conditions, none of them heart failure, and the headline trial had 32 patients.”
“I’m more convinced it will happen soon,” he said of widespread autonomous clinical AI. “Whether it reaches the people it’s meant for is a different question, and the first round of money was all sent to well-resourced academic and commercial sites. That’s fixable, but we need to put a safety-net system in the trial, enroll the patients the program says it’s for, and validate it on them first.”
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.