Thursday, February 20, 2014

Computer woes hit Banner hospital system: Another large EHR outage ... but patient safety was not compromised

Here is yet another story in the genre of "EHRs go out, but patient care has not been compromised." (See query link at http://hcrenewal.blogspot.com/search/label/Patient%20care%20has%20not%20been%20compromised; there are more than 20 posts there now):

Computer woes hit Banner hospital system
Ken Alltucker, The Arizona Republic 12:30 a.m. EST February 20, 2014
http://www.usatoday.com/story/news/nation/2014/02/19/computer-woes-slam-banner-hospital-system/5630829/

The Phoenix-based health system used backup paper records to help provide patient care.

PHOENIX -- Banner Health grappled with a widespread computer outage Wednesday as hospitals and doctors resorted to backup paper systems to provide care for patients.

The Phoenix-based health system did not immediately know what triggered the computer troubles that started just before 10 a.m. PST. An official described the computer troubles as a rolling outage of computer systems at hospitals and other health care facilities in Phoenix, Colorado and Nevada.

"Not knowing" means that you are not in control of your life-critical information systems; rather, they are in control of you.

By late Wednesday, a spokesman said, technicians had identified the problem and were fixing it. They expect to investigate the root cause of the problem Thursday.

It took from 10 AM to "late Wednesday" to identify a problem causing a mass outage.  That should give anyone pause about dependency on fragile information systems in the hands of hospital IT departments (whose personnel undergo an ocean's less qualification-vetting than the medical personnel who depend on their work product) for one's medical care.

Banner Health, the Phoenix area's largest health care system, activated "downtime procedures" that included using paper-based systems to track medications and other care provided to patients, officials said.

Banner's emergency departments still provided care to patients and accepted new patients.

Some non-emergency surgeries and appointments were delayed because of the computer troubles.

"There have been some delays and inconveniences, but we are still providing care," said Bill Byron, Banner Health's senior vice president of public relations.

In other words, what they are saying is "we really don't need these systems, that cost hundreds of millions of dollars, to provide care with the same degree of safety as with our 'downtime procedures' (a.k.a. paper)" ... and that patient safety was not compromised by this mass outage.

Banner Health, which operates 24 hospitals and several primary-care offices and outpatient centers in more than a half-dozen states, was working to "reboot" the computer systems Wednesday evening through a series of sequential fixes, Byron said.

In the meantime, Banner officials were able to retrieve computer-based records that detailed patients' medical histories, including any medications, laboratory results and procedures that were previously performed.

Officials?  What about line clinicians?  And when did this capability start if the systems needed to be "rebooted?"

Nurses and doctors shifted to writing on paper records after the computer systems experienced trouble Wednesday morning.

Information from those paper charts will be keyed into the patients' computer-based health records after the problem is fixed.

Sure, and nothing will be lost that could adversely affect patients in the future....

Banner Health has been among the most advanced health systems in the nation in converting to computer-based health records.

Banner Estrella Medical Center was among the first hospitals to open as an "all-digital" facility in the past decade. Banner's other hospitals have largely completed the final stages of installing computerized record-keeping in areas such as physician order and entry and electronic documentation.

If they are the most advanced, what does this event say about those less advanced?

Arizona law does not require hospitals to notify state health regulators in the event of such a widespread outage.

Health IT, as usual, enjoys widespread and extraordinary regulatory accommodation.

However, some hospitals have internal policies requiring that they notify health accrediting organizations or federal regulatory agencies, such as the Centers for Medicare and Medicaid Services, an Arizona Department of Health Services spokeswoman said.

And how many do?  Not many, I predict.

-- SS

11 comments:

kel said...

Perhaps "patient safety was not compromised," but was the security of patient identity maintained during the "rolling outage of computer systems"?

Was one or more of the Banner systems hacked? Has Banner identified its HIT vulnerabilities for data breach or theft?

Banner was furiously "spinning" the incident.

Thanks for your insights.

InformaticsMD said...

Patient safety was compromised on its face. The term "EHR" etc. are anachronisms. What we have now are enterprise command-and-control systems. When they go down...there is mass disruption of people, and increased risk.

-- SS

Anonymous said...

Sounds like a network outage, so leave it to Cisco to figure out what happened.
The comment about IT staff being so much less qualified to support systems than the clinicians that use the systems is just not fair. These folks are seasoned, qualified and highly trained individuals. Sure we don't go to school for "15" years, like some doctors do, but then we don't suffer from the god complex surely depicted in the comment.

Anonymous said...

Patients lives are cut short because of the delays in care wrought by such crashes. Having expereinced these crashes, there is a deadly silence in the hospital patient care as the screens go blank and all records vanish.

They have nothing, NOTHING. They have no idea what medicines the patient got and when. The have no odea where the patients are unpess someone had printed a list before the crash. Patients lie in a state of neglect as the mayhem abounds while no one know anything about how long the mayhem will last.

I would not want to be receiving nocare in a hospital when these devices crash.

Undoubtedly, there are or will be deaths in these hospitals from the crash.

Anonymous said...

Twas interesting that the news article did not mention the vendor. Wait no longer
https://www.hospitalmedicine.org/AM/Glycemic/pdf/Banner_Good_Samaritan_Medical_Center_Cerner_Screenshots.pdf

https://www.cerner.com/About_Cerner/Awards/HIMSS_Stage_6_and_7/

http://phx.corporate-ir.net/phoenix.zhtml?c=118401&p=irol-newsArticle&ID=765389&highlight

Steve Lucas said...

Was it a rolling failure or did some remote part of the system fail causing a cascade effect? With such a large system a node failure causing the system to fail is no system at all.

Steve Lucas

Anonymous said...

This, and other cases, is justification for there to be a National HIT Safety Investigation Board, not unlike for highways and avaiation and other transportation modes.

InformaticsMD said...

Steve Lucas said...

Was it a rolling failure or did some remote part of the system fail causing a cascade effect? With such a large system a node failure causing the system to fail is no system at all.

I have had the distinct displeasure of seeing systems designed - for God's sake - for ICUs that would crash if any one workstation crashed.

Some system, indeed. I wrote about the experience here: http://cci.drexel.edu/faculty/ssilverstein/cases/?loc=cases&sloc=clinical%20computing%20problems%20in%20ICU

-- SS

InformaticsMD said...

Anonymous February 21, 2014 at 5:10:00 PM EST said...

This, and other cases, is justification for there to be a National HIT Safety Investigation Board, not unlike for highways and avaiation and other transportation modes.

Such as public transit, in which I was a medical manager regarding safety-critical roles. When accidents happened...they were carefully investigated and the results made public, not brushed under the rug.

Of course a transit accident is more visible than injured or dead patients here and there.

-- SS

InformaticsMD said...

Anonymous 7:17:00 PM EST wrote:

Sounds like a network outage, so leave it to Cisco to figure out what happened.

That is perverse - let someone else fix the problem because we can't.

When in medicine, when I was faced with - for instance - a person with intestinal tuberculosis who was having a massive GI bleed in the middle of the night, with infectious, liquid black stool passing from his rectum, I didn't say "it's a bowel problem - that's outside my knowledge - let the GI doctors fix it."

I did what was needed ... then called in the cavalry after the patient was out of danger. My career would have been short indeed if I had.

Either you're in control of your information systems, or they are in control of you. Passing the buck means, on its face, you (departmentally speaking) are unqualified to be the steward of life-critical technology.

The comment about IT staff being so much less qualified to support systems than the clinicians that use the systems is just not fair. These folks are seasoned, qualified and highly trained individuals. Sure we don't go to school for "15" years, like some doctors do, but then we don't suffer from the god complex surely depicted in the comment.

Surely not.

I am both a physician and a computer person. I provide graduate instruction to future healthcare computing personnel. I've been a hiring manager of information scientists and IT personnel as well.

My experience in hospital IT departments has been disappointing; one of poor vetting and poor qualifications for the job of health IT anything - with a few exceptions, I would not have hired them.

(Business computing is another matter. Clinical computing and business computing are very different subspecialties. The latter optimally requires special education and competencies, in fact, per HHS (http://hcrenewal.blogspot.com/2009/12/onc-defines-taxonomy-of-health-it.html).

In hospitals, to get my projects accomplished & to remediate failing ones that were putting patients in danger I often had to replace the assigned personnel and hire my own (fortunately I was able to do so after securing that authority from the powers that be).

In pharma, the quality of the research/scientific IT personnel (as opposed to business IT) was much higher in terms of skills and expertise compared to hospitals' clinical computing personnel. This may be a factor of the more attractive pay and benefits offered by pharma.

Making matters worse, a decade ago, several dominant healthcare IS Recruitment firms published bizarre views on healthcare IS talent, views that most clinicians will not and cannot identify with. "I don't think a degree gets you anything," said healthcare recruiter Lion Goodman, president of the Goodman Group in California about CIO's and other healthcare IS staffers. Healthcare IS recruiter Betsy Hersher of Hersher Associates agreed, stating "There's nothing like the school of Hard Knocks." In seeking out CIO talent, recruiter Lion Goodman didn't think "clinical experience yields [hospital] IT people who have broad enough perspective. Physicians in particular make poor choices for CIOs. They don't think of the business issues at hand because they're consumed with patient care issues," according to Goodman.

I'm sorry, bizarre was too kind a word. Perverse is more accurate. I wonder how many of today's hospital IT mid- and top- level managers were hired on such criteria.

Finally, all IT personnel are not equal (just as all doctors are not equal).

(continued below)

InformaticsMD said...

(continued from above)

As author Bob Lewis points out in his book "IS Survival Guide", three decades ago Harold Sackman researched the performance gap between programmers. He found that the best ones were able to write programs 16 times faster and debug then 28 times faster than those created by "average" programmers, and when they were done their programs were six times more compact and ran five times faster.)

In a field as critical and complex as medicine, the need for star performers is especially acute.

Informaticists who know both medicine and computing are often especially good at identifying such computer professional qualities relevant to clinical medicine, for example through direct interviews.

This rarely happens in hospital IT shops.

-- SS