Saturday, June 06, 2009

Indiana Power Surge: The Hospital Computing Backwater And The Need to Reform IT Before IT Reforms Medicine

In the three part series "Healthcare IT Backwater: The $20 Billion Abyss?" I lamented that hospitals are often an IT backwater, and the backwater mentality would severely hamper efforts to "reform healthcare" via IT:
Merriam-Webster:

Main Entry: back·wa·ter           Listen to the pronunciation of backwater
Pronunciation: \-ˌwȯ-tər, -ˌwä-\
Function: noun
Date: 1629
- an isolated or backward place or condition

Computing is not a hospitals' core competency, salaries for IT personnel are usually lower than in other industries, and talent mismanagement in hospitals is far worse than in other sectors where IT is critical (e.g., health IT recruiters retained by hospitals have traditionally been generalists, or worse, recruiters with attitudes such as this):

I don't think a degree gets you anything," says healthcare recruiter Lion Goodman, president of the Goodman Group in San Rafael, California about CIO's and other healthcare MIS staffers. Healthcare MIS recruiter Betsy Hersher of Hersher Associates, Northbrook, Illinois, agreed, stating "There's nothing like the school of hard knocks." In seeking out CIO talent, recruiter Lion Goodman "doesn't think clinical experience yields [hospital] IT people who have broad enough perspective. Physicians in particular make poor choices for CIOs. They don't think of the business issues at hand because they're consumed with patient care issues," according to Goodman.


The end result of the hospital IT backwater mentality is stories such as this:

Indystar.com

Hospital is forced to turn away patients

Methodist Hospital went "on diversion" early Tuesday for the first time in its 100-plus years, sending ambulances that came to its doors to other hospitals.The diversion lasted until about 3 a.m., Wide said.

A power surge knocked out Clarian Health's computer system Monday afternoon, derailing the hospitals' ability to access electronic health records for patients, said Clarian spokesman James Wide. Staff members at Methodist and Indiana University Hospital had to enter patients' records by hand.

By about 1 a.m. Tuesday, a backlog of paperwork led Methodist and IU hospitals to stop accepting patients who arrived by ambulance. Walk-in patients were still accepted. The diversion lasted until about 3 a.m., Wide said.

This raises a number of questions:

  • Which power system "surged?" Shouldn't hospitals be protected from such "surges?"
  • Was any other equipment "knocked out?" CCU/ICU monitors? Ventilators? Heart-lung machines? If so, why were they not protected? If they were not knocked out but the EHR was, why?
  • What computers, exactly, were knocked out? Does not sound like the workstations, which would be rebooted quickly. Therefore must have been data center servers...and if so...
  • Why were the EHR servers vulnerable to "power surges"?
  • Or was this EHR a remote hosted ASP arrangement and did the surge occur at the service provider...in which case, the above question also applies?
  • Where were the backup systems? Were there backup systems?
  • Who performed the "backlog of paperwork?" Was it clinicians backloading the paper data into the system? -- in which case a power surge would have turned doctors into the world's most highly trained data entry clerks?

Of course, there was the usual refrain by non-clinical management, as I wrote at "Health IT Failure Never Puts Patients at Risk", that such outages are just minor blips on the radar. In another account of this story:

WTHR.com Eyewitness News

Power surge forces hospitals to divert new patients

Updated: June 2, 2009 07:57 AM

Indianapolis - Methodist and Indiana University Hospitals are back open and accepting new patients.

Monday around 3:45pm a massive power surge hit the hospitals crashing their computer system. IPL responded quickly to assist with bringing the system back online.

In the meantime, the hospitals began doing things on paper, like patient charting, that had been done by computer.

There was no disruption in patient care, however, around 1am Tuesday both hospitals decided to stop taking new patients [that's not a "disruption in patient care?" - ed.]

As of 3:15am most of the primary care system was back up and running and Methodist starting taking some new patients. I.U. Hospital continued to divert patients for some time.

As of 7:00am all patient primary care systems were up and running and both hospitals were completely open to new patients. Officials say there will be no impact on elective surgery Tuesday.


Here is the candid translation of the statement that "There was no disruption in patient care" by a total EHR crash, making records and probably CPOE unavailable and forcing clinicians and administrative assistants back to paper and pencil (fortunately, paper charts and pencils require at most only a flashlight and hand-cranked sharpener for read and write access):

"By the grace of God, no patients were harmed (that we know about in the confusion right now, or will admit to even if we did know) among our own patients or those diverted, in the chaos that ensued after the HIT systems went down from a power surge that we should not have been susceptible to in the first place."

As I wrote in Part 2 of the aforementioned series, I reiterate that HIT should be treated as experimental, not as a drop-in panacea for healthcare's ills. In its present state is is perhaps as likely to exacerbate those ills. This is probably not technology that should be deployed en masse at present. We cannot afford as a society to learn how to do this by trial and error.

-- SS

1 comment:

Irrision said...

I agree 100% with your statement that health care is an IT backwater. I work at a metro hospital as part of the core infrastructure support team. I can tell you a slew of reasons why it is that way but the main two causes I see are these:

1) Complete lack of funding. When we survey organizations with similar income profiles to our organization outside of health care, they universally spend 3-5 times more on infrastructure (servers, data center power/ac, and IT security).

2) I would take some issue with your statement about a lack of college education being the main cause of problems. It is more around the problem with not offering competitive pay to entice talented people (College Degree or not). This combined with an early tendency of hospitals recruit their IT staff from their medical staff and place them into IT management rather than hiring from outside. Nothing personal against medical staff but this would be similar to me trying to treat patients (or manage people who treat patients).

On your statement about surge suppression; Most of these organizations are probably operating systems in 25+ year old data centers that lack power feed redundancy (among many other things). If you take a lightening strike down your only feed, it will go down, it is supposed to take a hit and protect the equipment connected to the power distribution system.

That being said it is more likely that the 'power surge' was a something along the lines of a UPS failure or human error with a convenient white washing by management for the press.