Showing posts with label core incompetence. Show all posts
Showing posts with label core incompetence. Show all posts

Monday, January 15, 2018

"Hot Spot For User Entry Error": Hawaii missile alert: How one employee ‘pushed the wrong button’ and caused a wave of panic

A short post.

I believe this WaPo story vividly demonstrates issues I've seen in what Australian colleague Dr. Jon Patrick & I call "bad health IT." 

We came up with the simple-to-comprehend terminology "Good health IT/Bad health IT" in his living room in Sydney after my presentation to the Health Informatics Society of Australia in 2012 on health IT trust (http://hcrenewal.blogspot.com/2012/08/my-presentation-to-health-informatics.html), to replace my earlier terms "health IT done well" vs. "done poorly."

It was not just "one employee who pushed the wrong button."  A team of apparently incompetent IT personnel and utterly incompetent IT managers - completely devoid of any understanding of human-computer interaction - were, in essence, standing behind this this employee and guiding his hand.

The Hawaii mishap vividly demonstrates bad IT in the most critical of settings - badly conceived, designed & implemented, lacking appropriate safeguards, usually by people who do not know the domain, and often who are, dare I say, lacking common sense.

Questions that the incident raises include:
  • How [in God's name] were such critical items as "Test missile alert" and "Missile alert" (the real thing) residing in the same menu?  Who came up with such bad, terse labeling as well?  [A "hot spot" for user entry error] 
  • Why were there no reasonable safeguards? 
  • Why was it easy for anyone to make a big mistake?
  • Why was no system in place for rapid retraction?

The answers translate back to - well, I probably don't need to say it.




Hawaii missile alert: How one employee ‘pushed the wrong button’ and caused a wave of panic
Washington Post

... Around 8:05 a.m., the Hawaii emergency employee initiated the internal test, according to a timeline released by the state. From a drop-down menu on a computer program, he saw two options: "Test missile alert" and "Missile alert."

This is  a classic example of what can be called a "hot spot for user entry error."

... He was supposed to choose the former; as much of the world now knows, he chose the latter, an initiation of a real-life missile alert.

... "Based on the information we have collected so far, it appears that the government of Hawaii did not have reasonable safeguards or process controls in place to prevent the transmission of a false alert," Pai said in a statement.

... Part of what worsened the situation Saturday was that there was no system in place at the state emergency agency for correcting the error, Rapoza said...."In the past there was no cancellation button. There was no false alarm button at all,"

.... "Part of the problem was it was too easy - for anyone - to make such a big mistake," Rapoza said. "We have to make sure that we're not looking for retribution, but we should be fixing the problems in the system.

It would not be unreasonable to predict that U.S. armed forces were put on alert, perhaps even scrambling fighter planes near the Korean peninsula - moves that other countries could detect.

This "mishap" could have caused N. Korea or other hostile country to react, and led to catastrophe.




The utterly incompetent IT personnel and their utterly incompetent managers who birthed such a cornucopia of IT atrocities should be severely punished.

-- SS

Jan. 16, 2018.  Update, update.  Who's got the button?

This new WaPo story shows the "unholy" menu - a jumbled mess.


https://www.washingtonpost.com/news/morning-mix/wp/2018/01/16/that-was-no-wrong-button-in-hawaii-take-a-look/?utm_term=.147cc4852c8e


Was this designed by an expert in human-computer interaction?  I think not...


Incredible.

-- SS


Wednesday, June 19, 2013

"Computer problems" force docs back to paper charts at Memorial Hospital - From June 11 until at least June 24?

More on the wonders and dependability of commercial health IT, as implemented in hospitals, which are generally an IT backwaters:



(Illinois) — Computer problems have caused Memorial Hospital (http://www.memhosp.com/Pages/Home.aspx) staff back to the old days of using paper to chart patients' treatments.

The hospital's computer system, Meditech, went down late on Tuesday, June 11 and is not expected to be fully restored until at least Monday, June 24, according to the hospital. In the meantime, staff will be recording patients' medical records unto traditional paper charts.

"While the duration of this down time is unfortunate, all hospital services continue utilizing the backup process we have in place for occasions such as this," Memorial President Mark Turner said in a prepared statement. "I am extremely proud of the way our employees and medical staff have pulled together to maintain quality care and patient safety."

Once the system is up and running again, the information from the paper charts will be transferred into each patient's electronic medical record. The same standards for patient confidentiality and safety are being met, according to the hospital.

The computer problems are believed to result from upgrades added to the system in preparation for a major upgrade in July.

"The duration of this down time is unfortunate?"  This mission-critical system went down June 11 and won't be fully restored until June 24?  How is this even possible?  What, exactly, were their maximal-uptime, redundancy, disaster recovery and business continuity strategies?  Was there a natural disaster in that area I don't know about?

Regarding the hospital President's statement that:

"I am extremely proud of the way our employees and medical staff have pulled together to maintain quality care and patient safety."

I would translate that to read:

"Our IT incompetence is all on you, clinicians.  You're liable for any patient harm that results in the messy transitions from IT to paper, and back from paper to IT."

-- SS

Addendum - a question I've asked before - if everything is just fine, business as usual, with no safety impediments after reverting to paper due to emergency ... then why spend $100 million+ on EHRs?

Read more here: http://www.bnd.com/2013/06/17/2660472/computer-problems-force-docs-back.html#storylink=cpyA

Saturday, April 28, 2012

Don't Worry, Your Records are Safe - Part IV

At past posts "Don't Worry, Your Electronic Medical Records Are Getting Safer With Every Passing Day", "Another Episode of "But Don't Worry, Your Records are Safe..." and "Still More Electronic Medical Data Chaos, Pandemonium, Bedlam, Tumult and Maelstrom: But Don't Worry, Your Data is Secure", I wrote on the issue of medical record security.

Security from prying eyes, that is.

I didn't include security of data from placement into /dev/null (that is, destruction).

There's this email, received by East coast physicians not long ago from a claims processing company (identities redacted):

Dear Provider,

As you may be aware, we experienced a significant problem with our computer system during a software maintenance function on XX/XX/2010.

In addition to the network issue, we discovered that the redundant back-up systems were not operating as reported.  ["Reported" when, and by whom, one wonders? - ed.]

We had two on-site back-up systems that were monitored daily and which were historically reported as successful.  We have since learned that these internal back-up functions were not operating as reported and the on-site back-ups were not entirely successful. [Meaning, they were not successful, period - ed.]

Also, our software vendor, [major EHR vendor], was providing two additional remote back-ups on servers located in [city, state] and [city, state]. [EHR vendor] has informed us that these remote back-ups were not initiated as represented.  [Meaning, they screwed up - ed.]  Therefore, when our computer network system malfunctioned, there was no readily available back-up data on-site or at the remote redundant back-up servers.

Please be aware that we have replaced hardware components and were able to recreate the data bases and we are billing.  However, we are still unable to access data that was stored on our servers prior to XX/XX/2010.

[EHR vendor] is diligently working to retrieve the data from the hard drives, back-up tapes, and through other means.  Please be assured that all files will be restored, if the files cannot be fully restored electronically, then they will be fully restored manually.

At [our claims processing company], we are truly saddened by the fact that we have disappointed clients and we sincerely apologize for any inconvenience experienced by you, your staff, or your patients.

We have always appreciated your loyalty as a valued client and will continue to keep you informed of the progress.

The levels of information technology and data management incompetence exhibited in this message are stunning. 

The confidence it imparts regarding the safety of our critical medical data from destruction, and its availability when truly needed, is less than stellar.

A major problem is that the health IT industry has no accountability. 

I believe the Food, Drug and Cosmetic Act needs to be amended to become the "Food, Drug, Cosmetic, and Cybernetic" Act.

-- SS

Saturday, November 19, 2011

If gov't officials can't deal with bendy bananas, curvy cucumbers and water for hydration, how can they possibly consider issues with computers?

I present this story with almost no commentary except this post's title, the simple observation that keeping government out of many aspects of out lives is not a bad idea, and a few minor points:

The Telegraph

EU bans claim that water can prevent dehydration

Brussels bureaucrats were ridiculed yesterday after banning drink manufacturers from claiming that water can prevent dehydration.

By

6:20AM GMT 18 Nov 2011

EU officials concluded that, following a three-year investigation, there was no evidence to prove the previously undisputed fact.

Producers of bottled water are now forbidden by law from making the claim and will face a two-year jail sentence if they defy the edict, which comes into force in the UK next month.

Last night, critics claimed the EU was at odds with both science and common sense. Conservative MEP Roger Helmer said: “This is stupidity writ large.

“The euro is burning, the EU is falling apart and yet here they are: highly-paid, highly-pensioned officials worrying about the obvious qualities of water and trying to deny us the right to say what is patently true.

“If ever there were an episode which demonstrates the folly of the great European project then this is it.”

NHS health guidelines state clearly that drinking water helps avoid dehydration, and that Britons should drink at least 1.2 litres per day.

The Department for Health disputed the wisdom of the new law. A spokesman said: “Of course water hydrates. While we support the EU in preventing false claims about products, we need to exercise common sense as far as possible."

German professors Dr Andreas Hahn and Dr Moritz Hagenmeyer, who advise food manufacturers on how to advertise their products, asked the European Commission if the claim could be made on labels.

They compiled what they assumed was an uncontroversial statement in order to test new laws which allow products to claim they can reduce the risk of disease, subject to EU approval.

They applied for the right to state that “regular consumption of significant amounts of water can reduce the risk of development of dehydration” as well as preventing a decrease in performance.

However, last February, the European Food Standards Authority (EFSA) refused to approve the statement.

A meeting of 21 scientists in Parma, Italy, concluded that reduced water content in the body was a symptom of dehydration and not something that drinking water could subsequently control.

Now the EFSA verdict has been turned into an EU directive which was issued on Wednesday.

Ukip MEP Paul Nuttall said the ruling made the “bendy banana law” look “positively sane”.

He said: “I had to read this four or five times before I believed it. It is a perfect example of what Brussels does best. Spend three years, with 20 separate pieces of correspondence before summoning 21 professors to Parma where they decide with great solemnity that drinking water cannot be sold as a way to combat dehydration.

“Then they make this judgment law and make it clear that if anybody dares sell water claiming that it is effective against dehydration they could get into serious legal bother.

EU regulations, which aim to uphold food standards across member states, are frequently criticised.

Rules banning bent bananas and curved cucumbers [what?? - ed.] were scrapped in 2008 after causing international ridicule.

Prof Hahn, from the Institute for Food Science and Human Nutrition at Hanover Leibniz University, said the European Commission had made another mistake with its latest ruling.

“What is our reaction to the outcome? Let us put it this way: We are neither surprised nor delighted.

“The European Commission is wrong; it should have authorised the claim. That should be more than clear to anyone who has consumed water in the past, and who has not? We fear there is something wrong in the state of Europe.”

Prof Brian Ratcliffe, spokesman for the Nutrition Society, said dehydration was usually caused by a clinical condition [like hot weather and low fluid intake? - ed.] and that one could remain adequately hydrated without drinking water. [As the study was done in Italy, perhaps they had something else in mind? - ed.]

Water? Where we're going, we don't need water.

He said: “The EU is saying that this does not reduce the risk of dehydration and that is correct.

“This claim is trying to imply that there is something special about bottled water which is not a reasonable claim.”


Hard as this is to believe, the ruling is no April Fool's joke.

-- SS

Wednesday, August 04, 2010

GE: Don't Know Much About Radiation Safety, Don't Know Much About Physics

Don't know much about history

Don't know much biology
Don't know much about a science book
Don't know much about the french I took.
(Wonderful World, sung by Sam Cook)

This is becoming the theme song for executives of health care corporations.  We have posted about a series of cases in which major health care corporations suddenly seemed unable to carry out their core business functions, a phenomenon I am going to start calling "core business incompetence."  Some recent examples:
-  Baxter International apparently failed to check the purity of heparin it bought from a foreign supplier; the contaminated heparin resulted in approximately 81 deaths. (See post here.)
-  A major Genzyme manufacturing facility had multiple quality problems, resulting in the production of a very expensive, but contaminated biologic medication. (See post here.)
-  Three Johnson and Johnson manufacturing facilities had multiple quality problems, resulting in the production of contaminated batches of over-the-counter medications for children. (See post here.)
-  Aetna made mathematical errors in computing the rates it proposed charging; WellPoint made computer errors that exposed sensitive policy-holder information.  (See blog post here.)

Manufacturing pure, unadulterated medicines is a core function of a drug company.  Correctly calculating policy-costs and correctly handling patient data are core functions of health insurance companies.  But in the above cases, three large pharmaceutical companies and two large insurance companies could not perform these core functions competently.

Radiation Sickness from CT Scans
In the last few days, more details of what appears to be another example of core incompetency afflicting a major health care organization has surfaced.  The details were reported by Walt Bogdanich in the New York Times. Here is the summary:
When Alain Reyes’s hair suddenly fell out in a freakish band circling his head, he was not the only one worried about his health. His co-workers at a shipping company avoided him, and his boss sent him home, fearing he had a contagious disease.

Only later would Mr. Reyes learn what had caused him so much physical and emotional grief: he had received a radiation overdose during a test for a stroke at a hospital in Glendale, Calif.

Other patients getting the procedure, called a CT brain perfusion scan, were being overdosed, too — 37 of them just up the freeway at Providence Saint Joseph Medical Center in Burbank, 269 more at the renowned Cedars-Sinai Medical Center in Los Angeles and dozens more at a hospital in Huntsville, Ala.

The overdoses, which began to emerge late last summer, set off an investigation by the Food and Drug Administration into why patients tested with this complex yet lightly regulated technology were bombarded with excessive radiation. After 10 months, the agency has yet to provide a final report on what it found.

But an examination by The New York Times has found that radiation overdoses were larger and more widespread than previously known, that patients have reported symptoms considerably more serious than losing their hair, and that experts say they may face long-term risks of cancer and brain damage.

So not to mince words, in this sad case, many patients seem to have been subject to radiation poisoning from a diagnostic x-ray procedure (not from radiation therapy for cancer.)  Most people (and physicians) may have not previously thought that radiation poisoning was a possible harm from a single diagnostic CT scan.

A Design Failure?

Moreover, it appears that the radiation poisoning did not result from some freak accidents or malfunction:
The review also offers insight into the way many of the overdoses occurred. While in some cases technicians did not know how to properly administer the test, interviews with hospital officials and a review of public records raise new questions about the role of manufacturers, including how well they design their software and equipment and train those who use them.

Also,
None of the overdoses can be attributed to malfunctions of the CT scanners, government officials say.

At Glendale Adventist Medical Center, where Mr. Reyes and nine others were overdosed, employees told state investigators that they consulted with GE last year when instituting a new procedure to get quicker images of blood flow, state records show. But employees still made mistakes.

As a result, hospital officials said, a feature that technicians thought would lower radiation levels actually raised them. Cedars-Sinai gave a similar explanation.

'There was a lot of trust in the manufacturers and trust in the technology that this type of equipment in this day and age would not allow you to get more radiation than was absolutely necessary,' said Robert Marchuck, the Glendale hospital’s vice president of ancillary services.

A GE spokesman, Arvind Gopalratnam, said the way scanners were programmed was 'determined by the user and not the manufacturer.' GE, he added, has no record of Glendale seeking its help setting up the new procedure in 2009.

Most of the known overdoses, including the biggest, occurred on scanners made by GE Healthcare. At two hospitals that use Toshiba scanners — Los Angeles County-U.S.C. and South Lake in Florida — officials said the manufacturer suggested machine settings that ultimately produced too much radiation. Representatives of Toshiba agreed to be interviewed in their California office but abruptly canceled.

In particular,
To this day, no one at Cedars-Sinai knows who programmed the scanners that delivered the overdoses, officials there say. But in written statements to The Times, hospital officials said they had figured out how they might have occurred.

Normally, the more radiation a CT scan uses, the better the image. But amid concerns that patients are getting more radiation than necessary, the medical community has embraced the idea of using only enough to obtain an image sufficient for diagnosis.

To do that, GE offers a feature on its CT scanner that can automatically adjust the dose according to a patient’s size and body part. It is, a GE manual says, 'a technical innovation that significantly reduces radiation dose.'

At Cedars-Sinai and Glendale Adventist, technicians used the automatic feature — rather than a fixed, predetermined radiation level — for their brain perfusion scans.

But a surprise awaited them: when used with certain machine settings that govern image clarity, the automatic feature did not reduce the dose — it raised it.

As a result, patients at Cedars-Sinai received up to eight times as much radiation as necessary, while the 10 overradiated at Glendale received four times as much, state records show.

GE says the hospitals should have known how to safely use the automatic feature. Besides, GE said, the feature had 'limited utility' for a perfusion scan because the test targets one specific area of the brain, rather than body parts of varying thickness. In addition, experts say high-clarity images are not needed to track blood flow in the brain.

GE further faulted hospital technologists for failing to notice dosing levels on their treatment screens.

But representatives of both hospitals said GE trainers never fully explained the automatic feature.

In a statement, Cedars-Sinai said that during multiple training visits, GE never mentioned the 'counterintuitive' nature of a feature that promises to lower radiation but ends up raising it. The hospital also said user manuals never pointed out that the automatic feature was of limited value for perfusion scans.

A better-designed CT scanner, safety experts say, might have prevented the overdoses by alerting operators, or simply shutting down, when doses reached dangerous levels.

Summary: Core Incompetence
To summarize, GE diagnostic CT scan machines apparently were designed so that they could deliver so much radiation that their use could cause radiation sickness. The machines had no built in safeguards to limit the dose of radiation. Users, that is, X-ray technicians, could program the machine so as to deliver dangerous radiation doses, but GE did not warn them that this was the case, nor provide a machine feature that would give a real-time warning of incipient overdose.

Thus it appears that GE, which has been described as "the world’s biggest maker of health care imaging and information technology systems," failed in its core function of designing and manufacturing safe diagnostic x-ray machines. (Presumably the same might be said of Toshiba, but the NY Times report does not include so much detail about problems with its machines.)

A straightforward explanation of the magnitude of the core incompetence was provided by one of the patients who got radiation sickness:
To Mr. Heuser, it is unconscionable that equipment able to deliver such high radiation doses lacks stronger safety features.

'When you are in a car and it backs up, it goes beep, beep, beep,' he said. 'If you fill the washing machine up too much, it won’t work. There is no red light that says you are overradiating.'

It really seems like we are seeing an epidemic of core incompetence by major US health care corporations.

I can only speculate about why this occurring. One explanation is that the organizations may be lead by people who do not understand the health care context, and do not understand the scientific, engineering, and technical issues involved with core competence. Many health care corporations have come to be lead by people with no experience or background in relevant areas.

For example, the current, and often acclaimed CEO of GE is Jeffrey Immelt, who "earned a B.A. degree in applied mathematics from Dartmouth College in 1978 and an M.B.A. from Harvard University in 1982." The leader of GE Healthcare is John Dineen, who "is a graduate of the University of Vermont where he earned Bachelor's degrees in biology/genetics and computer science," and "joined GE in 1986 as a telecommunications engineer."

Another, even more speculative explanation is that many major US and global health care organizations seem to have been infected with the virus of putting short-term financial results ahead of everything else, leading to cutting of costs and particularly human expertise in the areas most core to the organizations' functions, and pushing for hurried results even at the expense of clear thinking, carefully engineering, and consideration of the effects on patients and other humans of the resulting mistakes.

Hopefully, further investigation will reveal more about what went wrong in this case. Meanwhile, I say again, again, again,...

As long as "imperial CEOs" can continue to get extremely rich while presiding over incompetence and stupidity, if not worse), we can expect the foolishness to continue. Meanwhile, the foolishness drives up costs and drives down quality of health care for the poor suffering patients, let alone the physicians and other health care professionals who must deal with it.

To really reform health care, we need to provide incentives for competent, honest leadership, and make that leadership accountable for its shortcomings.

Postscript - Why the Rush to Aggressive Treatment?

The NY Times article described the rush to get one of the afflicted patients to the CT scan:
on the morning of July 4, when a 52-year-old executive producer of films, H. Michael Heuser, arrived in the emergency department with stroke symptoms.

A 'code brain' was immediately called, signaling a life-or-death situation. A blood clot in the brain can be dissolved with medicine, but doctors must do it within several hours, before brain cells die from a lack of oxygen. So Mr. Heuser was rushed into a room with several CT scanners, where he underwent one brain perfusion study and at least one more later. A CT perfusion scan, which lasts about 45 seconds, can identify a stroke through a series of blood flow images.

Mr. Heuser did have a stroke, from which he would recover. But other parts of his body inexplicably began to break down.

However, there is no clear evidence that such a rapid and drastic approach is really that good for patients. A recently updated Cochrane review concluded: "In patients with acute ischaemic stroke, immediate anticoagulant therapy is not associated with net short or long-term benefit. Treatment with anticoagulants reduced recurrent stroke, deep vein thrombosis and pulmonary embolism, but increased bleeding risk. The data do not support the routine use of any the currently available anticoagulants in acute ischaemic stroke." However, many physicians seem to advocate the drastic approach based on some controlled trials that showed at best small improvements in average neurological function for patients receiving rapid anti-coagulation (and which were done before anyone thought about radiation sickness as a possible harm of the approach.) Once again, an aggressive, technological, expensive approach, possibly advocated by people who stood to gain financially from it, may not be as advantageous as it appears.