Why does healthcare tolerate atrocious up-time?
Last week’s CrowdStrike outage crippled banking, government agencies, airlines and hospitals. The outage was so bad, the Department of Transportation has now opened a formal investigation into Delta. While airline delays can mean a few rough days, hospital outages are a matter of life and death: surgeries stop, electronic medical records with allergies and crucial past history are inaccessible, emergency rooms stop taking ambulances. The toll is arguably far worse.
So why are airline delays getting all the press coverage? In part, because hospitals tolerate terrible software by adapting to electronic unreliability. They practice “down time” the way elementary schools run fire drills–by breaking out the paper charts, calling in extra administrative help, and generally adopting an operational mindset of treading water until the software is back up.
Hospital software is largely an oligopoly, dominated by a few major electronic health record (EHR) companies–one of which has attained more than half of the market share. Leaders in these corporations were heavily involved in writing the Obama-era “meaningful use” legislation designed to encourage EHR adoption. The hope and hype that EHRs would lead to improved safety, efficiency, and patient access to their data has largely been disappointing. And without federal incentives there is basically no innovation. EHRs are basically a commodity–a static industry–more focused on killing competition than achieving the goal that the government paid for them to help achieve.
The largest EHR vendors have technology stacks dating back to the late 1970s. This is software that runs exclusively on Windows. Remember Windows 98 software? For those outside healthcare, that’s the look, feel, and overwhelming complexity of a “modern” EHR which hospitals spend billions on. It’s been widely covered that doctors spend two hours in electronic medical records for every hour they spend with patients.
As bad as EHR software is from a usability and quality of life perspective, there’s one aspect no one seems to mention: it’s also terribly unreliable. Modern cloud-based software strives for “five 9s” reliability: up 99.999% of the time. That means it’s “down” (inaccessible) for about 5 minutes each year.
By contrast, EHR software follows the same ancient paradigm as those “critical security updates” you did back in the 90s for your Windows 95 desktop. It’s routine for the emergency room - where lives are on the line - to be without the EHR for several hours each month during such “upgrades”
Why do we stand for this? Better alternatives exist. Vital, which is a cloud based guide to emergency & hospital stays used by ~3M patients each year, had no downtime during CrowdStrike. Nor any significant downtime in recent memory (see status.vital.io if you’d like).
Cloud-based software allows modern software development. Vital actually pushes new code to production 10 to 20 times a day, just with good test coverage. We use a “blue-green” server architecture. When version 137 of the software is deployed on blue servers, a router instantly switches all the traffic from v136 software over. If there’s a failure, it instantly switches back. Both versions are running simultaneously for a time, allowing zero downtime.
This is a radical departure from the old-school Windows .exe package install system. Such systems are vulnerable to CrowdStrike-like errors. This should be a wake-up call for healthcare to get off old desktop-Windows based EHRs.
The solution is clear: health systems must move away from desktop Windows’s based systems. They should adopt cloud-native software - not 1990s style thick-client software merely on someone else’s server. That’s paying lip-service to “cloud”. Your EHR - and every healthcare IT vendor you work with - should be accessible in any web browser, and from any mobile device. It should use a secure, multi-tenanted architecture, and practice continuous upgrades with zero downtime. Today’s EHRs are maybe 97% reliable. That’s more than a week of cumulative down-time each year. Lives are at stake, and your clunky EHR is one of the biggest reasons hospitals have trouble retaining staff. Your patients and staff deserve better.