Blog
The Hearth: How to Make AI Safe Enough for the Patient Journey
Fire, left uncontrolled, burns down houses. Contained in a hearth, it warms a family and feeds them. AI in healthcare is the same. The technology is powerful and it isn’t going away. The real work is building the structure around it that turns raw capability into something safe.
That challenge brought three leaders together for a fireside chat on AI and the patient experience at HMPS26, the Healthcare Marketing and Physician Strategies Summit, in Salt Lake City in May 2026: Aaron Patzer, founder of Vital; Dr. Bridget Duffy, a longtime patient-experience leader and the nation’s first CXO; and Dr. Raj Ratwani, a human-factors and patient-safety scientist and VP of Scientific Affairs at MedStar Health. From very different vantage points, they kept returning to the same idea: AI should restore the human parts of care, not crowd them out.
Patients aren’t waiting for permission
That structure matters now because patients are already using AI for their care. They’re pasting lab results, discharge instructions, and biopsy reports into general-purpose chatbots, because their own records are written in a language they can’t read. Many of those tools are trained on the open internet rather than the medical literature, and as Patzer warns, AI “is so good at creating the most confident and happy bad answer that you can ever get.”
When a health system is slow to offer something better, it doesn’t prevent AI use. It simply hands patients to the riskiest option available.
The goal is more human interaction, not less
The way forward is not to push more technology between clinicians and patients. It’s to use technology to clear the busywork that pulls them apart. As Dr. Duffy puts it, “if we could just get the stupid stuff off the plates of doctors and nurses, the empathy can come from humans if the technology enables it.” The point, Patzer adds, is “to have more human interaction, not less.”
That matters because most healing has little to do with the machine. Dr. Duffy often recalls a mentor’s lesson that “only 20% of healing is linked to the technology,” and that the other 80% comes from treating people as whole human beings. Good AI protects that 80%.
“If we could just get the stupid stuff off the plates of doctors and nurses, the empathy can come from humans if the technology enables it.”
What safe AI actually requires
Doing this safely takes real engineering, not good intentions. A few principles consistently separate trustworthy systems from risky ones:
Train on quality sources. Use the peer-reviewed literature, not internet forums.
Have one AI check another. “You need one AI to judge the other,” Patzer explains, comparing a plain-language summary against the original record, an approach that “takes care of about 80% of your errors.”
Surface the model’s confidence and act on it. Don’t treat a guess and a certainty the same way.
Always disclose when a patient is talking to AI. Trust depends on it.
Keep a clinician in the loop for anything sensitive. Data alone is never the whole story. As Dr. Duffy notes, a seasoned clinician still knows a child is truly sick “when the patient has lost the twinkle in their eye.”
Stop evaluating the same tool ten times over
There’s a smarter way to evaluate these tools, too. The burden of vetting AI has landed on provider organizations, many evaluating the same systems independently at great cost. Dr. Ratwani frames it plainly: no pilot or airline audits the software in the cockpit, “it’s done by the manufacturer,” against a shared standard.
Healthcare deserves the same. A free, industry-built framework for grading patient-facing AI, much like a nutrition label, lets teams assess a tool in about half an hour.
Building toward a shared standard
The goal, in Dr. Ratwani’s words, is “human AI teaming,” where the human trusts the AI and the AI complements the human. Done well, that future is proactive rather than reactive. Instead of patients pulling answers out of a chatbot on their own, a health system monitors what someone actually needs and reaches out first, watching out for them rather than waiting for a crisis.
Getting there is less about any single product and more about the standard the industry chooses to hold. Adopters set that bar. When health systems refuse to deploy unvetted tools and demand the same evidence from every vendor, the market follows. When they move too slowly, that decision gets made for them, by the consumer chatbots patients are already turning to.
So the practical starting point is simple. Define the specific problem you’re trying to solve before you shop for tools. Insist on apples-to-apples comparisons, whether through a shared framework or a model card that works like a nutrition label. And measure what actually matters: not just efficiency, but whether the technology gave clinicians more time with their patients.
As Dr. Duffy says, “There should be no competition around innovation that improves humanity.” Build the hearth, and the fire stops being something to fear. It becomes the thing that keeps everyone warm.