Another day, another EHR outage: MEDSTAR EHR goes dark for days

At my March 2, 2015 post "Rideout Hospital, California: CEO Pinocchio on quality of patient care during hospital computer crash" (http://hcrenewal.blogspot.com/2015/03/rideout-hospital-california-ceo.html) I highlighted a stunning example of when the light shone through the corporate B.S. about health IT outages, thanks to a letter to the editor by a family member of an affected patient:

Letter: Re: Rideout Hospital computer problems

http://www.appeal-democrat.com/opinion/letter-re-rideout-computer-problems/article_4a408cc0-be47-11e4-9b7b-93c22da930d4.html 

Friday, February 27, 2015 

I am writing in regard to comments made by the CEO of Rideout Hospital regarding its recent computer crash. 

He said quality of care for patients had not been compromised during this incident. He is lying.

My spouse went to Rideout almost two weeks ago and had a Lexiscan of her heart when the computer system went down. The hospital doctor released her and assured her that if anything were wrong, the radiology department would spot it and she would inform us.

Here it is two weeks later and now they are saying because of the computer problem the entire test didn't get to her cardiologist until today. They think she may have had a minor heart attack and needs further cardiac intervention.

 Is this the new "open and improved" truths we are getting from this hospital? Rideout CEO Robert Chason misinformed us all. 

I am sure my spouse, who has fallen through the cracks during this inexcusable lapse in Rideout's technical policies, is not the only patient suffering similar situations. 

Shame on Chason for minimizing the effects of this catastrophe at our local hospital. 

Edward Ferreira 
Yuba City

I am aware of another major EHR outage via Politico.com:

4/9/15
http://www.politico.com/morningehealth/0415/morningehealth17818.html

MEDSTAR EHR GOES DARK FOR DAYS: MedStar’s outpatient clinics in the D.C. and Baltimore area lost access to their EHRs Monday and Tuesday when the GE Centricity EHR system crashed. The system went offline for scheduled maintenance on Friday and had come back on Monday when it suffered a “severe” malfunction, according to an email from Medstar management that was shared with Morning eHealth.

“All of a sudden the screens lit up with a giant text warning telling us to log off immediately,” a doctor said. “They kept saying it would be back up in an hour, but when I left work Tuesday night it was still down.”

This doctor told us that the outage was “disruptive and liberating at the same time. I wrote prescriptions on a pad for two days instead of clicking 13 times to send an e-script. And I got to talk to my patients much more than I usually do.

But of course we didn’t have access to any notes or medication history, and that was problematic.” MedStar notified clinicians in the email that any information entered in the EHR after Friday was lost.

I do not know if corporate issued the standard "patient safety was not compromised" line, but can almost predict it was uttered somewhere along the line.

MedStar is a big healthcare system.  An outage for several days at its outpatient clinics is disruptive and will lead to harms in the short term, but also in the long term, that cannot be effectively tallied, due to lost information. 

That includes information put on backup paper that fails to get entered when an EHR goes back up, as well as outright computer data loss as occurred here.

Note the doctor's comments about the "liberating" aspect of being freed from health IT.  He/she could actually practice medicine, not computer babysitting.

How many harms will come of this "major malfunction?"  There is no way to know.  However, hospitals cannot have it both ways.  If these systems are touted as improving safety, then safety is affected when they are down and emergency measures are put into place, resulting in chaos; and certainly when information simply goes to the "bit bucket."

The answer?  Either far more redundancy, or far less reliance on "paperless" systems.

There also needs to be mandatory reporting of EHR outages and root cause analysis so the incidence and the reasons can be studied, at the very least.

-- SS

Rideout Hospital, California: CEO Pinocchio on quality of patient care during hospital computer crash

EHRs and other clinical IT are touted as essential to improving safety, among many other benefits.

Yet when hospital systems crash, the common refrain by hospital executives to the press, when such stories are reported, is "...but quality of care was not compromised."  

In fact, I've made an indexing term for this refrain.  The following query link retrieves the posts so indexed, numbering almost 30 at present:  http://hcrenewal.blogspot.com/search/label/Patient%20care%20has%20not%20been%20compromised

One hospital in California 40 minutes north of Sacramento had a crash and its CEO made exactly that claim.  However, a patient's husband disagreed, and called the CEO a liar.  Why?  His wife was affected by the crash in a very unsafe manner.

The Appeal-Democrat is a local news source for Sutter and Yuba counties, California, serving readers since 1860.  Emphases mine:

Letter: Re: Rideout Hospital computer problems

http://www.appeal-democrat.com/opinion/letter-re-rideout-computer-problems/article_4a408cc0-be47-11e4-9b7b-93c22da930d4.html 

Friday, February 27, 2015 

I am writing in regard to comments made by the CEO of Rideout Hospital regarding its recent computer crash. 

He said quality of care for patients had not been compromised during this incident. He is lying.

My spouse went to Rideout almost two weeks ago and had a Lexiscan of her heart when the computer system went down. The hospital doctor released her and assured her that if anything were wrong, the radiology department would spot it and she would inform us.

Here it is two weeks later and now they are saying because of the computer problem the entire test didn't get to her cardiologist until today. They think she may have had a minor heart attack and needs further cardiac intervention.

 Is this the new "open and improved" truths we are getting from this hospital? Rideout CEO Robert Chason misinformed us all. 

I am sure my spouse, who has fallen through the cracks during this inexcusable lapse in Rideout's technical policies, is not the only patient suffering similar situations. 

Shame on Chason for minimizing the effects of this catastrophe at our local hospital. 

Edward Ferreira 
Yuba City

Claims that hospital paralysis through health IT outages and malfunctions don't compromise patient care insult my intelligence.  Such claims insult the intelligence of patients and their families, too.  Outages and malfunctions nearly always compromise the quality and safety of care.
  
Patient safety is put at risk because hospitals are not making adequate efforts to keep these systems up 24x7. Many might say they can't afford it.  You don't put in life-critical information systems half-baked, however. Not in medicine, anyway.

Finally, the press, by accepting these Pinocchio-like statements from hospital administrations without severe challenge, only promote cavalier behavior of hospital executives.

Hospital executives:  EHRs are so absolutely essential to patient safety, we spend hundreds of millions of dollars on them. When they crash, however, patient care is never compromised.

-- SS

What in God's Name is Going on With Healthcare IT at Cambridge University Hospitals?

This story about a  UK hospital that recently "went live"with an American electronic health record/enterprise command-and-control system (EPIC) was not only predictable, but expected considering the sorry state of the health IT industry in terms of clinical leadership and regulation.

(It appears this was a "big bang" rollout, see http://www.ehi.co.uk/news/EHI/8845/cambridge-goes-for-epic-big-bang, an implementation method better suited for warehouses and widget suppliers than major hospitals.)

Addenbrooke’s staff blame blood shortage on new eHospital
By CambridgeNews  |  Posted: November 05, 2014
http://www.cambridge-news.co.uk/Addenbrooke-8217-s-staff-blame-blood-shortage-new/story-24513716-detail/story.html

By Freya Leng

Members of staff at Addenbrooke's [hospital, http://www.cuh.org.uk/addenbrookes-hospital] have voiced their concerns about the new IT system which has been blamed for a blood shortage.

Cambridge University Hospitals' eHospital went live on October 26 and is designed to improve the quality of care for patients by allowing clinicians and frontline staff to access patient information wherever they are, at the click of a button.

I must put to rest this lie once again.  The unregulated, generally terrible software being sold by the so-called EHR vendors is NOT simply software to allow clinicians to  "access patient information wherever they are."  

This is enterprise clinician and clinical resource command-and-control software, through which increasingly each transaction related to care must pass.  In other words, ERP packages to manage patient care, as one might manage inventory and shipping in a merchant enterprise:

http://en.wikipedia.org/wiki/Enterprise_resource_planning

Enterprise resource planning (ERP) is a business management software—usually a suite of integrated applications—that a company can use to collect, store, manage and interpret data from many business activities

Unfortunately, the reductionist assumptions behind the conception, design, authoring and implementation of such ERP software - that hospitals and healthcare are linear, predictable processes - are both deadly wrong, and the beliefs of fools and the recklessly cavalier.

To wit:

But since the launch, the News has been contacted by a senior member of staff at Addenbrooke's who said the new IT system was having "serious consequences" on the "operational running of the service".

In a letter, the staff member who does not want to be named [due to potential for retaliation - ed.], said: "The hospital has very little blood available due to transfusion lab technical failures. Truth - the new IT system is responsible."

The letter also states the impact the shortage of blood has had on the hospital including the cancellation of all elective surgery until November 8 as well as impacting on any procedure that holds a risk of blood transfusion and organ transplantation.

"I believe sufficient risk has been placed upon all patients under care of Addenbrooke's," the staff member said. "Someone needs to be responsible for the implementation of the new IT system."

In my experience, the non-clinical executives who often select this technology, and the IT personnel who then implement the technology (often ignoring clinicians), do need to be held responsible for bad outcomes - in the courtroom.

The News also understands the whole system went down for six hours at the weekend with staff reverting back to paper and all major trauma cases diverted elsewhere.

There are numerous cases on this blog of disruptive and patient-endangering EHR system outages.  These are simply inexcusable regarding life-critical computing. The unexpected transitions back and forth between paper endanger patients.

A doctor, who also did not want to be named, said the wifi system which supports all the ward rounds is "unfit for purpose" and is leading to gross inefficiency.

If this is true, it once again represents the cavalier nature of those technologists rarely held accountable for mistakes that, of they occurred in other critical industries (e.g., aviation, nuclear energy) might leave smoldering ruins and radioactive clouds that would result in the end of their careers...at the very least.  Unfortunately, individual injured and dead patients are not quite as visible to the public.

"The general feeling on the ground is that they could not have implemented the system any worse than they have done and without any doubt it has already significantly affected patient care." they said.

This is consistent with my own personal experience with hospital IT departments in the U.S., where mistakes that I could not even have conceived of making, were regularly made - leaving me to have to point out and clean up the mess, at risk to my own career due to the reactions of the non-clinical IT leaders and staff to being shown their own inadequacy regarding clinical affairs.  (This was, of course, an odd reaction by people who'd never gone to medical school, let alone had doctoral or postdoctoral study, research and development experience in Medical Informatics.)

A CUH spokesman said: "eHospital gives our staff more time with patients at the bedside, many of whom are frail, elderly and have complex conditions.

Right, just those patients who are most vulnerable to IT debacles and the cascading errors that can result.

"However, unlike banks, shops or travel agents, we cannot close our doors or stop our services to the hundreds of thousands of people we treat every year. So it was always going to be a challenge to implement such a massive change.

That is a very poor excuse for IT malpractice.  It makes the reader believe everything possible in due diligence was done, that others' experience was completely paid attention to, etc.  The results give me great doubt about that...

"Pathology was affected early last week, which led to a brief reduction in the number of tests, but we are increasingly operating as normal. We did carry out a successful 're-boot' of the system early on Sunday morning.

There we go once again  the typical bureaucratic spin that "the malfunctions were minor, nothing to see here, move along, patient safety was not compromised"  (a recurrent refrain with its own index term on this blog, see the 25+ posts at http://hcrenewal.blogspot.com/search/label/Patient%20care%20has%20not%20been%20compromised) - while at the same time these systems are represented as revolutionizing medicine - except when they malfunction, at which time they have no meaningful effects on care.

"The much bigger challenge we face is that the Trust is incredibly busy and we have limited numbers of beds available, and which need to be kept free for emergency cases. Operations will continue to be rescheduled until the community care for those who no longer need a hospital bed is in place. We do sympathise with the frustration that people feel and apologise for the delay they are experiencing."

In my opinion, patients put at risk, and injured and dead patients need and deserve more than apologies for information technology malpractice.

Especially at Cambridge University, where in my opinion, this whole affair is truly a world-class embarrassment.

-- SS

Full Moon Friday the 13th: Computers down at Vermont's largest hospital

A health IT "glitch" hit in Vermont on Full Moon Friday the 13th (have some werewolf with your bad luck!)

Dark Shadows' Quentin Collins (David Selby) in a foul full-moon mood.

Note, however, that "bad luck" has little or nothing to do with mission-critical healthcare IT mass outages.  Mismanagement often does:

http://www.wcax.com/story/25773022/computers-down-at-vermonts-largest-hospital

Computers down at Vermont's largest hospital

Posted: Jun 13, 2014 4:12 PM EST Updated: Jun 13, 2014 4:12 PM EST

By WCAX News

BURLINGTON, Vt. -
A major, systemwide computer problem at Fletcher Allen Health Care in Burlington Friday.

The hospital says the computer system went down at 9:45 a.m. The glitch means doctors and nurses could not enter patient information into computers.

"Glitch" is an unfortunate euphemism for potentially harmful or deadly computer malfunctions, often due to mismanagement of that technology. See query link http://hcrenewal.blogspot.com/search/label/glitch.

Hospital spokesperson Mike Noble says information is now being recorded manually [i.e., on paper - ed.] Noble says the staff continues to provide the same care as normal and no surgeries were canceled because of the computer issue.

Noble says the issue appears to be an internal problem and not something from an outside computer provider.

It’s not known when the problem will be corrected.

This is yet another "the systems that revolutionize healthcare against all of paper's ills have all gone down, we don't know when they will be fixed, but patient care has not been compromised because we are resorting to paper" story (more at query link http://hcrenewal.blogspot.com/search/label/Patient%20care%20has%20not%20been%20compromised). 

-- SS

Canada: Province-wide electronic medical record computer system 'glitch' causing patients to be turned away from care

This story describes a very bad scenario for sick Canadians.  I offer just a few pithy comments, as not much more than that is needed:

http://medicinehatnews.com/news/local-news/2014/06/10/system-failure-has-docs-patients-upset/

System failure has docs, patients upset

By Gillian Slade on June 10, 2014

Many patients were turned away from their doctor’s office Monday because a province-wide electronic medical record computer system had collapsed.

Province-wide.  Stunning.  A very big argument against centralization of EHR resources.

“This is the third straight week of issues with the TELUS Wolf system,” said Dr. Donovan Nunweiler at Southlands Medical Clinic. “We feel we were encouraged by government to switch to Wolf and now it’s not working.”

I wonder when someone in the Canadian government is  going to issue that now-famous slogan "but patient safety has not been compromised"...

A year ago 202 physician clinics across Alberta using TELUS Wolf were unable to access patient records for most of the day.

More than 200 physician clinics are blind, deaf and dumb?  Wonder what happens to acute patients on days like that.

On Monday patients arrived only to be told the electronic patient files were not accessible making it impossible to see test results, past medical history and medications.

Paper never goes on strike.  Perhaps elimination of paper completely is not such a good idea?

“This is affecting me big time and affecting my income,” said Ken Hoeppner, a patient at HealthWORX Medical clinic, who had waited 15 days for his appointment. “Every time government touches something they wreck it. Our health care used to be good here before Alberta Health Services took over.”

No comment.

At HealthWORX, office manager Carel Liebenberg said the office was doing what it could to reschedule people. One patient had driven three hours to be there for his appointment early Monday.

It's just a "glitch", sir or madam.  Stop complaining. (http://hcrenewal.blogspot.com/search/label/glitch)

At Health Matters Medical Clinic, staff confirmed they too were dealing with no access to patients’ records on TELUS’s Wolf system.

Originally to encourage physicians to move to electronic medical records, the government gave a monetary incentive. Alberta Health selected TELUS Health Solutions Wolf EMR after a request for proposals in 2008.

There was a requirement for the service to be available 99.9 per cent of the time between 6 a.m. and midnight with financial penalties for failure to do so.

“There is no longer any government support,” said Nunweiler. “We (Southlands Medical Clinic) pay $2,000 a month for this. Who is going to hold TELUS accountable now? The government has abandoned us. Cost and issues switching patient data, when systems are not compatible, prevents us from going somewhere else.”

Seems to be this TELUS:  https://www.telushealth.com/health-solutions/electronic-health-records-%28ehr%29.  Sounds like a monopoly to me.

On that page:

TELUS Electronic Health Records (EHR) provides a better way to share, access and consolidate information.

Without quick, secure access to complete and reliable information, healthcare provision can be inefficient, preventing patients from receiving the best care possible.

How ironic.

Liebenberg reached TELUS at 9 a.m. Monday.

“They said they had just become aware of the issue and that their data technicians were in a meeting discussing the problem,” said Liebenberg. “Last week the system was extremely slow, taking 15 minutes for a physician to simply renew a prescription.”

Data technicians were in a meeting?  Sounds like a fantastic way to respond to a Province-wide medical emergency.

Nunweiler said he’d made notes on paper as he struggled to manage the snail’s pace of the system last week. Monday he would be adding to those notes and envisaged several hours at night entering the data to make it current.

Paper never goes on strike.

Liebenberg said the need to re-schedule appointments reflects badly on the clinic and some patients don’t understand it’s a system failure beyond the clinic’s.

That's just great for patient-physician relations.

Dr. Franz Yonker said HealthWORX had been using JonokeMed but the government endorsed TELUS Wolf and physicians were encouraged to switch.

“I think this is really bad for a government-backed system,” said Yonker.

"Wolf" is a somewhat humorous name considering these problems.  As in, a predatory EHR ... one wonders just how much better the others are.

Becky Nelson arrived for her appointment to refill prescriptions and was concerned about how long it would take to get another appointment.

“The government needs to get this on track. We are suffering the consequences,” said Nelson.

I wonder if any patients will suffer the ultimate consequence.  (Hint to Canadians:  never become too dependent on the Government.  Stuff like this happens.)

Donna Schneider brought her mother Vernie Ferguson in for results of some tests. Ferguson said she was not at all well.

“There is nowhere else to go and get my test results,” said Ferguson.

Held medical hostage to bad health IT.  How horrible.

The News requested an interview with TELUS but there was no response on Monday afternoon.

The News requested information from Alberta Health but that was not available on Monday.

Perhaps they're busy, in meetings discussing how to fix the problem.

-- SS

Computer woes hit Banner hospital system: Another large EHR outage ... but patient safety was not compromised

Here is yet another story in the genre of "EHRs go out, but patient care has not been compromised." (See query link at http://hcrenewal.blogspot.com/search/label/Patient%20care%20has%20not%20been%20compromised; there are more than 20 posts there now):

Computer woes hit Banner hospital system
Ken Alltucker, The Arizona Republic 12:30 a.m. EST February 20, 2014
http://www.usatoday.com/story/news/nation/2014/02/19/computer-woes-slam-banner-hospital-system/5630829/

The Phoenix-based health system used backup paper records to help provide patient care.

PHOENIX -- Banner Health grappled with a widespread computer outage Wednesday as hospitals and doctors resorted to backup paper systems to provide care for patients.

The Phoenix-based health system did not immediately know what triggered the computer troubles that started just before 10 a.m. PST. An official described the computer troubles as a rolling outage of computer systems at hospitals and other health care facilities in Phoenix, Colorado and Nevada.

"Not knowing" means that you are not in control of your life-critical information systems; rather, they are in control of you.

By late Wednesday, a spokesman said, technicians had identified the problem and were fixing it. They expect to investigate the root cause of the problem Thursday.

It took from 10 AM to "late Wednesday" to identify a problem causing a mass outage.  That should give anyone pause about dependency on fragile information systems in the hands of hospital IT departments (whose personnel undergo an ocean's less qualification-vetting than the medical personnel who depend on their work product) for one's medical care.

Banner Health, the Phoenix area's largest health care system, activated "downtime procedures" that included using paper-based systems to track medications and other care provided to patients, officials said.

Banner's emergency departments still provided care to patients and accepted new patients.

Some non-emergency surgeries and appointments were delayed because of the computer troubles.

"There have been some delays and inconveniences, but we are still providing care," said Bill Byron, Banner Health's senior vice president of public relations.

In other words, what they are saying is "we really don't need these systems, that cost hundreds of millions of dollars, to provide care with the same degree of safety as with our 'downtime procedures' (a.k.a. paper)" ... and that patient safety was not compromised by this mass outage.

Banner Health, which operates 24 hospitals and several primary-care offices and outpatient centers in more than a half-dozen states, was working to "reboot" the computer systems Wednesday evening through a series of sequential fixes, Byron said.

In the meantime, Banner officials were able to retrieve computer-based records that detailed patients' medical histories, including any medications, laboratory results and procedures that were previously performed.

Officials?  What about line clinicians?  And when did this capability start if the systems needed to be "rebooted?"

Nurses and doctors shifted to writing on paper records after the computer systems experienced trouble Wednesday morning.

Information from those paper charts will be keyed into the patients' computer-based health records after the problem is fixed.

Sure, and nothing will be lost that could adversely affect patients in the future....

Banner Health has been among the most advanced health systems in the nation in converting to computer-based health records.

Banner Estrella Medical Center was among the first hospitals to open as an "all-digital" facility in the past decade. Banner's other hospitals have largely completed the final stages of installing computerized record-keeping in areas such as physician order and entry and electronic documentation.

If they are the most advanced, what does this event say about those less advanced?

Arizona law does not require hospitals to notify state health regulators in the event of such a widespread outage.

Health IT, as usual, enjoys widespread and extraordinary regulatory accommodation.

However, some hospitals have internal policies requiring that they notify health accrediting organizations or federal regulatory agencies, such as the Centers for Medicare and Medicaid Services, an Arizona Department of Health Services spokeswoman said.

And how many do?  Not many, I predict.

-- SS

Another in the "Health IT Crashed, But Patient Care Was Not Compromised" Series - NHS Greater Glasgow and Clyde

We must have health IT to prevent stupid doctors and nurses from making mistakes.  Without it, patients are at the mercy of dreaded Paper Records, which by definition cannot ever be used to provide quality care.

But when the IT goes down, Patient Care Has Not Been Compromised.  This line should be trademarked, as it's seen so often.  (I even have an indexing tag for it, see this query link: http://hcrenewal.blogspot.com/search/label/Patient%20care%20has%20not%20been%20compromised.)

Care is never, never compromised when the IT goes belly up en masse due to information technology malpractice.  Care is only compromised by paper, no matter how good the paper records and its human stewards are.

Here's the latest example that made it to the news, in Scotland:

1 October 2013
BBC News
Appointments postponed after major IT failure at NHSGGC (NHS Greater Glasgow and Clyde)

Hundreds of outpatient appointments and a number of operations had to be postponed after computer systems failed at Scotland's biggest health board.

NHS Greater Glasgow and Clyde said technicians were working through the night to fix a "major IT problem" which occurred on Tuesday morning.

It affected staff access to clinical and administrative systems.

Delaying hundreds of appointments and delaying surgeries at up to 10 major hospitals seems on its face to represent "compromised care."

The health board apologised to patients and said all appointments would be rescheduled.

In total, 288 outpatient appointments, four planned inpatient procedures, 23 day surgery cases and 40 chemotherapy sessions were postponed.

There was also some delay in calls to the switchboard being answered.

The problem may have affected up to 10 major hospitals across the health board area.

One wonders how these appointments and surgeries were triaged for delay.  Clearly the downed computer was of no help.

Here's that wonderful line:

But emergency operations were not compromised - neither were community services.

So when does computer failure actually compromise patient care?  I'd like to see some hospital executive with a spine for once admit that IT malpractice does disrupt patient care, create distractions, and thus create safety risk.  Considering the domain, however, I doubt I'll ever see that.

A spokeswoman said: "Our technical staff are working flat out to resolve this.

It should never have happened in the first place.

"The problem relates to our networks and the way staff can connect to some of our clinical and administrative systems.

Well, sick patients really appreciate that explanation.

It was not clear how long the disruption would last.

NHSGGC said if it did continue, people who were scheduled for treatment would be contacted directly.

Per a computer guru from my past:  "Either you're in control of your information systems, or they're in control of you."

In this instance, the latter clearly applies.

In healthcare, having your information systems in control of you is, sooner or later, going to be deadly.

-- SS

Setback for Sutter after $1B EHR crashes (in followup to post "RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals")

At my July 12, 2013 post "RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals" (http://hcrenewal.blogspot.com/2013/07/rns-say-sutters-new-electronic-system.html) I reproduced a California Nurses Association warning about rollout of an EHR at Sutter:

RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals

Introduction of a new electronic medical records system at Sutter corporation East Bay hospitals has produced multiple problems with safe care delivery that has put patients at risk, charged the California Nurses Association today.

Problems with technology are not unique to health care ...  [What is unique to healthcare IT is the complete lack of regulation - ed.]

In over 100 reports submitted by RNs at Alta Bates Summit Medical Center facilities in Berkeley and Oakland, nurses cited a variety of serious problems with the new system, known as Epic. The reports are in union forms RNs submit to management documenting assignments they believe to be unsafe.

Patient care concerns included computerized delays in timely administration of medications and contact with physicians, ability to properly monitor patients, and other delays in treatment.  Many noted that the excessive amount of time required to interact with the computer system, inputting and accessing data, sharply cuts down on time they can spend with patients with frequent complaints from patients about not seeing their RN.  [Note: patients are not given the opportunity for informed consent about the risks, nor opt-out of EHR use in their care - ed.]

In related posts I'd observed such concerns being ignored by hospital management.  See header of the aforementioned post.

Now we have this:  a major system crash.

Healthcare IT News
Setback for Sutter after $1B EHR crashes

'No access to medication orders, patient allergies and other information puts patients at serious risk'

SACRAMENTO, CA | August 28, 2013

The nearly $1 billion electronic health record system at Sutter Health in Northern California crashed early this week, leaving nurses and clinical staff unable to access any patient information for a full day.
he Epic EHR was down Aug. 23 due to a planned upgrade for eight hours during which nurses and other clinical users could read medication orders and patient histories but had to record new data on paper to then be re-entered into the system later, according to reports from the California Nurses Association, part of National Nurses United, the largest nurses union in the U.S.

Then, on the morning of Aug. 26 at approximately 8 a.m., the system crashed completely, at which time nurses, physicians and hospital staff had no access to patient information, including what medications patients were taking or required to take and all vital patient history data. 
 
The Epic EHR went dark at several Sutter Health locations, including Alta Bates Summit Medical Center, Eden Medical Center, Mills-Peninsula Hospital, Sutter Delta, Sutter Tracy, Sutter Modesto and numerous affiliated clinics.

"This incident is especially worrisome," said Bonnie Castilla, RN, CNA legislative director, in a news release. "No access to medication orders, patient allergies and other information puts patients at serious risk.  These systems should never be relied upon for protecting patients or assuring the delivery of the safest care."  [This is no exaggeration.  My mother was nearly killed by such a problem when caused by an EHR defect (link) that I averted at the last minute by happenstance of showing up at the right time, while she recuperating from another EHR-related disaster - ed.]   

Some RNs also reported that Sutter failed to provide adequate support during the blackout and didn't implement adequate backup planning.
 
"All information such as medication administration records and patient histories was outdated by two to three days," said Mike Hill, Alta Bates Summit RN, in a news release. "There were no orders that could be seen of any kind through the day so nurses called for what they needed."

Kevin Sweat, an emergency room RN at Eden Medical Center, a Sutter affiliate, also weighed in on the blackout, saying that in his 12 years as an RN he has never seen a system crash like this. 
  

Worse, clinicians must now serve their Cybernetic Master to perfection, or be whipped (apparently to improve morale):

... "We have been on Epic for 5 months now, and we can no longer have incorrect orders, missing information or incorrect or missing charges. Starting on September 1st, errors made in any of the above will result in progressive discipline," according to another hospital memo sent to staff.

In the setting of dire warnings by the nurses of EHR dangers several months back that were likely largely ignored, if any patient was harmed or killed as a result of this latest fiasco, the corporate leadership has literally begged to be sued for negligence, in my view.

However I'm sure a press release soon will claim that "patient care has not been compromised."

Of course this includes now and moving forward, even with informational gaps all over the place.

-- SS

Aug. 29, 2013 additional thought:

The punishment for not being a 'perfect' user of this EHR is the ultimate "blame the user" (blame the victim?) game, considering the pressures of patient care in hospitals in lean times - partly due to EHR expense! - and EHRs that have not been formally studied for usability and are poorly designed causing "use error" (that is, a poor user experience promotes even careful users to make errors).  Cf. definition of bad health IT:

Bad Health IT ("BHIT") is defined as IT that is ill-suited to purpose, hard to use, unreliable, loses data or provides incorrect data, causes cognitive overload, slows rather than facilitates users, lacks appropriate alerts, creates the need for hypervigilance (i.e., towards avoiding IT-related mishaps) that increases stress, is lacking in security, compromises patient privacy or otherwise demonstrates suboptimal design and/or implementation.

The study of usability is getting underway only now via NIST but will likely be done in an industry-friendly way due to health IT politics.

-- SS

Aug. 29, 2013 addendum

There have been numerous comments over at HisTalk (at http://histalk2.com/2013/08/27/news-82813/) defending the outage as not EPIC's fault.   From the point of view of clinicians - and more importantly, patients - it doesn't matter what component of the hospital's entire "EHR" (an anachronistic term used for what is now a complex enterprise clinical resource and clinician command-and-control system) went down. 

Aside from all the EPIC issues the nurses have been complaining about (see earlier July 12, 2013 post linked above), the larger problem is that IT malpractice occurred.  The term "malpractice" is used in medical mishaps; I see no reason why it does not apply to major outages of mission critical healthcare information technology systems.

IT malpractice in healthcare kills.

These are the types of nurses I'd want caring for me and mine.  Letting this kind of snafu go "anechoic" does not promote proper management remedial education on Safety 101 and on health IT risk, two areas of education that management appears to desperately need in hospitals.

-- SS

Boulder Community Hospital computer records back on line - but something does not add up

This post is in followup to my March 20, 2013 post "Boulder Community Hospital computer system crash: Either you're in control of your information systems, or they're in control of you".

At a March 24, 2013 Denver Post article "Boulder Community Hospital computer records back on line" the following statements are made:

The computer system that Boulder Community Hospital uses to manage patient records, which had been down for almost two weeks, is now up and running again, hospital officials said Saturday.

Meditech, the system used by the hospital to manage patient records, went down March 12 and affected the hospital, its Foothills campus, eight laboratories and six imaging centers. It was put back into full service at about 3 p.m. Friday, according to hospital spokesman Rich Sheehan.

Sheehan said an investigation showed the outage was a result of a malfunction in one of the main computer servers ... the hospital has replaced the hard drives for the server that failed and are inspecting the remaining servers ... [the failure] resulted in the system being unable to access patient information. The malfunction affected both the primary server and a backup server kept off-site.

A hard drive failure led to a two-week outage of an entire EHR system and its offsite backup server?  A mission-critical system in a hospital is so fragile that a hard drive failure caused a two week outage?

If so, that itself shows, at best, poor overall system design with regard to reliability and redundancy (any server worth its salt has hard drives in a failure-tolerant configuration e.g., RAID), but also is not quite credible on its face.  A remote server should not be taken down by the failure of a local server.  I suspect the failure was more than just a hard drive failure, including software bugs or configuration errors, mass hardware and/or network failure, or even sabotage.

The following statement also lacks believability on its face:

... All patient data was recovered except for an eight-hour period the day of the outage. Sheehan said the hospital had to re-create, re-enter and validate the patient information for that eight-hour period before the system could resume normal operations.

If an information system is down for two weeks, there's two weeks worth of data lost.

... Sheehan said the hospital has replaced the hard drives for the server that failed and are inspecting the remaining servers. The hospital is also now doing data backups every four hours as opposed to every six hours, and is planning on doing hourly backups by the end of the week.

Replacing a failed hard drive is an inadequate precaution.  A 'system redundancy makeover' seems in order for when the next hard drive fails.   Hard drives have a very well known MTBF (mean time between failure) and annual failure rate.  (The very Seagate ST3750528AS hard drive in the PC I am typing this blog post on has an Annualized Failure Rate of 0.34%, per the manufacturer's publicly-available literature.) 

... An independent consulting firm also has been hired to conduct an investigation. The hospital said it expects a report within a few weeks. 

As other organizations are using Meditech products, Joint Commission Safety Standards (as I wrote in a 2009 JAMA letter to the editor "Health Care Information Technology, Hospital Responsibilities, and Joint Commission Standards" available at this link) call for sharing the results of that report with other organizations.  I had discussed this letter numerous times with senior Joint Commission leadership.

Will sharing of the independent consultant firm's report happen?  Probably not.

However, rest assured the Plaintiff's attorneys of Colorado will request it in malpractice suits that arose during the time period of outage.

-- SS

Boulder Community Hospital computer system crash: Either you're in control of your information systems, or they're in control of you

Yet another health IT crash, "prolonged" this time, from some unspecified "glitch":

Boulder Community Hospital computer system crash frustrates patients
Officials say it could take until Friday for outage to be resolved
By Brittany Anas
Camera Staff Writer
Posted:   03/18/2013 07:23:23 PM MDT
Updated:   03/18/2013 07:24:16 PM MDT

A prolonged computer system outage is preventing Boulder Community Hospital from accessing patient records -- making it difficult for people to schedule surgeries, get test results and make appointments for routine blood work.

Meditech, the system used by the hospital to manage patient records, went down in the middle of last week. It could take the hospital until Friday to get the system back up, said Rich Sheehan, spokesman for Boulder Community.

That fits my definition of "prolonged."

While information technology officials are investigating what caused the outage, Sheehan said patient records are protected and hospital officials don't believe they've been hacked. 

That's not very reassuring, considering the length of the outage.

The outage affects the hospital, its Foothills campus, eight laboratories and six imaging centers. 

Patients are on its face put at-risk ... for example, I know of several deaths of infants and adults from delayed x-ray reports alone ... but the clinicians, not the IT seller or hospital IT staff, are liable.

"We know medical care is important to people, so we understand the concerns those in the community have," Sheehan said. "We have a lot of people working on this, doing the best they can to solve this problem in a safe manner and as quickly as possible." 

"We know medical care is important to people?"  No, really?

In the meantime, the hospital is using manual paper record-keeping systems and traditional paper charts for its inpatients. Hospital officials say the system allows them to continue treating patients, provide diagnostic services and collect important clinical information that will be entered later into each patient's electronic health record.

But that concerns Eroca Lowe, whose mother was in the hospital Thursday through Sunday with gallbladder pain.

Lowe said the outage made it extremely difficult for doctors and nurses to do their jobs while hunting down lab results. She criticizes the hospital for not having a backup computer system and resorting to paper records.

"That's not a hospital in 2013," she said.

It's a good bet the paper records and HIM personnel managing them are not what they used to be pre-computer.

... Dina Huber said it took her and her significant other six days to schedule an appointment for a hernia surgery because the system used for scheduling is down.

"If they can't keep their computer system running, how can we trust them to perform surgery?" Huber said.

Fortunately, surgeons perform surgery ... not computers, IT staff or management.  Doctors, as the enablers of healthcare, don't need computers to save lives.

However, making their job harder is not a good idea.

A physician who works at Boulder Community Hospital, speaking on the condition of anonymity, said he doesn't think the outage is compromising the health or safety of patients. But, he said, the backup response "seems a little haphazard, and it's not an organized plan." He said physicians are left chasing down records.

If a prolonged outage "is not compromising safety", then why did the hospital spend tens of millions on computers?

Sheehan said the hospital is prioritizing accuracy and patient safety while getting the records system up and running. 

Once it's running again, there is significant risk of data now recorded manually being lost, thus again increasing error risk.

"We apologize for the delays, but this was an unavoidable situation," Sheehan said.

("We apologize for the chilly water, but this was an unavoidable situation." - Captain of the HMS Titanic?)

If an injury occurs, how will that sound to a jury?

Let me answer that:  like bull***.  

My response to Mr. Sheehan and Meditech, and the IT personnel involved:  "Either you're in control of your information systems, or they're in control of you."

It seems the latter clearly applies here.

I pray nobody gets injured ... and that the principals don't end up before plaintiff attorneys I've educated on the issues of bad health IT.

-- SS

When "Human Error" Causes EHR Downtime, Who is Liable For Patient Injuries That Result?

In the Pittsburgh Post-Gazette was this story of yet another EHR "glitch":

March 9, 2013 12:17 am 

Human error the cause of UPMC electronic issue

A systemwide problem with UPMC's information systems Wednesday left electronic patient records and other data inaccessible for about three hours. A UPMC spokeswoman said the hospitals "immediately went to manual backup systems, and we quickly identified and fixed the problem." She said there was no indication that patient care was compromised by the incident, which was due to human error.

I will presume the "human error" was not a physician or nurse pressing the wrong button, but a "human error" involving the servers or IT infrastructure such as a botched system upgrade, action that caused a server room power fault, etc.

UMPC is a very large system as their webpage shows, showing approximately fifteen major facilities.

The now-expected "patient care has not been compromised" line was provided to the Gazette, a line so commonly heard after EHR outages that I  use it as a Healthcare Renewal indexing tag (see this query link).

The following questions arise:

• What, exactly, was the "human error" and why was there no fault tolerance built into these mission-critical systems to account and compensate for it, such as via redundancy?

• If paper is so bad as a record-keeping medium that hundreds of billions of dollars are being spent to replace it, then how can patient care not be compromised, especially when multiple hospitals unexpectedly and without warning have to return to its use? 

• How can a very large hospital system rapidly declare that "patient care was not compromised" without a thorough and comprehensive patient review, accounting for possible delayed negative outcomes (by way of just a few simple examples, due to medication or imaging delays?)

•  Who is liable for any adverse patient outcomes that occurred related to the sudden unavailability of past records:  the clinicians?  The "human" who committed the computer-related error?  The corporation, either for direct negligence in implementing and mandating use of a system prone to mass outage by human error, or vicariously for the negligence and/or misconduct of its information technology employees and/or agents?

•  How many "outages" will it take before some patient is outright, no-doubt-about-it harmed or killed?  Do we want to find out, or is a priority to have redundancies so these systems don't crash?

I, for one, would not want to have a family member be "crashing" at the time of a sudden, unexpected system-wide EHR outage.

-- SS