Monday, March 07, 2011

Australian ED EHR Study: An End to the Line "Your Evidence Is Anecdotal, Thus Worthless?"

At my Sept. 2010 post "The Dangers of Critical Thinking in A Politicized, Irrational Culture" I lamented that while my early mentor in biomedicine Victor P. Satinsky MD taught the wise credo "critical thinking always, or your patient's dead" in the 1970's, our culture had become so perverse that this credo had been largely supplanted with:

"Critical thinking anytime, and your career's dead."

That post was in reaction to continued heckling on a professional mailing list, the American Medical Informatics Association (AMIA) Clinical Information Systems Working Group (cis-wg). On this list, Oregon Health Sciences University professor William Hersh expounded on how the evidence of health IT dangers was largely "anecdotal" therefore to be discounted, and how I, specifically, "didn't know the literature on health IT."

In that post I put the lie to the latter figment. On the former prevarication Dr. Jon Patrick, author of the recent thorough dissection of problems with the ED EHR system being rolled out in public hospitals in New South Wales, Australia (see my Mar. 5, 2011 post "On an EMR Forensic Evaluation by Professor Jon Patrick from Down Under: More Thoughts"), hit the ball out of the park:

Prof. Patrick to AMIA cis-wg:

I think such defences are particularly unuseful especially with respect to the dismissal of personal stories and experiences as "anecdotes", hence committing them to the realm of folklore. I offer these notions as a counterpoint.

Discounting Anecdotes:

1. Is a perfidious and specious act.

2. It denies early warning signs of problems.

3. It denies a voice and disempowers the working clinical community who have to operationalise decisions made by others.

4. It denies a route to process improvement within an institution - which is most important for EBM and incremental review of local processes.

5. It defends software manufacturers from fault rectification - cuts off even a need to deliberate on it. Critics of the value of anecdotes are squarely on the side of the faulty and deficient manufacturer.

6. A rule of project management is that projects consist of 3 components, cost, quality and time and if their needs to be a compromise it has to be on quality. Anecdotes are early warning signs of such a compromise.

Prof. Patrick had to once again put the lie to this refrain at a comment on the HISTalk blog yesterday, where the anonymous proprietor had written in a review of Patrick's EHR study:

... On the other hand, I wouldn’t say it’s [Prof. Patrick's Cerner FirstNet study] necessarily unbiased, it focuses on implementation of a single department application that didn’t go well for a variety of reasons (despite many successful FirstNet implementations elsewhere), it uses the unchallenged anecdotal comments of unhappy users who make it clear they liked their previous EDIS better, and it nitpicks (I wasn’t moved to find a pitchfork when I learned that the primary keys in the Millennium database aren’t named consistently).

[Considering the complexity and changeability of healthcare and the corresponding software lifecycle, I duly note that that latter attitude about 'failing to find pitchforks' regarding breaches of sound software engineering practices seems to be a symptom of the larger health IT disease that Prof. Patrick writes about - ed.]

Dr. Patrick then knocked the ball out of the Southern Hemisphere with a comment of his own about anecdotalism and a link to an expansion of the aforementioned ideas he'd shared on "discounting anecdotes":

Prof. Patrick to HISTalk owner:

Your Comment “it uses the unchallenged anecdotal comments of unhappy users ” is not only unfair but unreasonably inaccurate. The comments made by the users are the Directors of 7 EDs and so they have a right to carry authority by virtue of the experience but also the number of 6 out 7 presenting a view of Firstnet as unfit for purpose has numeric validity, which they justify with about 20+ pages of their comments – see Part 2 Appendix 2.

I would also point readers to my editorial about the role of personal experiences being the most useful information to understand the nature of socio-technical failures. http://aci.schattauer.de/en/contents/archive/issue/1124/manuscript/15463/show.html

The essay at that link, "The Validity of Personal Experiences in Evaluating HIT", is an editorial in Johns Hopkins informaticist Chris Lehmann's brilliant new journal "Applied Clinical Informatics."

The editorial is available free, and is a must-read for anyone in a decision-making or managerial role in mission critical domains, including our elected representatives.

In the editorial Dr. Patrick concludes. similarly to his earlier AMIA mailing list opinion:

... the denial of recounted personal experiences in discussion and analysis of HIT is biased and specious and has the effect of:

1. Denying early warning signs of problems.

2. Denying a voice for the working clinical community who have to operationalise decisions made by others and thus disempowers them.

3. Denying process improvement within an institution – which is most important for Evidence Based Medicine and incremental review of local processes.

4. Discourages staff from engaging in any form of process improvement hence worsening the sense of disenchantment.

Every legitimate personal experience of a HIT deserves to be considered on its merits lest we wish to retreat from process and product improvement. Mechanisms of censorship both implicit due to contrived processes of disinformation and disempowerment or explicit due to contractual specifications will lead to more waste, lost productivity, contempt for the providers, and distress among frontline staff rather than increased productivity and improved patient health and safety as we all desire.

In my view, the drivers or motivators for the "anecdotalist" accusation are these, singly or in combination:

  • Too much "education" to see the nose on one's face, as in, to think zebras and unicorns instead of horses when hearing hoofbeats outside one's midwest U.S. abode (eggheads);
  • Too little common sense (fools), as in Scott Adams' example: "IGNORING ALL ANECDOTAL EVIDENCE - Example: I always get hives immediately after eating strawberries. But without a scientifically controlled experiment, it’s not reliable data. So I continue to eat strawberries every day, since I can’t tell if they cause hives";
  • Too much concern for the possible interruption of flow of money or power in one's direction (gonifs).

In conclusion, the anecdotalist refrain of "your evidence is anecdotal" [therefore of little or no value] when used repetitively against competent observers is the refrain of eggheads, fools and gonifs.

In healthcare, the end result is "your patient's dead."

My "anecdotal relative" injured in a mid-2010 HIT mishap is sadly an example.

[June 2011 addendum: my relative, after much suffering, has now died of complications of the "anecdotal HIT mishap" - ed.]

As for myself, I am a Markopolist (see my Sept. 2010 post "Health IT: On Anecdotalism and Totalitarianism").

-- SS

11 comments:

Bernard Carroll said...

My nephew is a physician ED Director in New South Wales. His reply when I consulted him about this story was "System is a shambles" and "As usual, the consumers (doctors) had virtually no input." How's that for an anecdote?

Keith said...

What strikes me is that there seems to be too little evidence either way to support or disprove the benefits of HIT. With large health care systems now using these systems, where is the data showint that the touted beneifits are real. It disturbs me that the reasons for roll out have been cost savings and quality improvemnt, but there is a dearth of data showing either is acheived. This give rise to the conspiracy theorists who see IT manfacturers overly influencing our policy makers or goverment wonks who hope to use the accumulated data in nefarious ways. The argument boils down to alot of anecdotal stories on both sides and little solid research to back up claims.

Scot M Silverstein MD said...

Keith anonymous,

See my "Updated reading list on health IT."

The list in my view certainly does not lend support national rollouts of commercial health IT at this point in time.

This give rise to the conspiracy theorists who see IT manfacturers overly influencing our policy makers or goverment wonks who hope to use the accumulated data in nefarious ways.

This could almost be taken as an industry PR/reputation management firm meme.

Have you considered the possibility, in fact, that you might be the "conspiracy theorist" regarding those leveling those suspicions?

CEO's of a HIT company hob-nobbing with the POTUS, pronouncements of certainty about HIT "revolutionizng medicine one thunderous mouse click at a time" from Secretaries of HHS, and HITECH acts appearing w/o public input are just several factoids that need to be considered before deciding who's the conspiracy theorist, and who's the realist regarding "anecdotes."

But Dr. Patrick's made that point already.

The argument boils down to a lot of anecdotal stories on both sides and little solid research to back up claims.

Perhaps, although the research in recent years seems to be leaning towards lack of efficacy and towards dangers (see my aforementioned updated reading list post, and my recent post on MAUDE accounts, for example).

Finally, if your comments about "anecdotal arguments on both sides" were true, considering the domain of human subjects research, it would call for an immediate cessation of large scale HIT rollout until the research could be conducted.

"First, do no harm" is the overriding ethical principle.

That point is a first principle and is not open to debate.

-- SS

Scot M Silverstein MD said...

Bernard Carroll wrote:

"As usual, the consumers (doctors) had virtually no input."

The UK's NPfIT in the NHS programme failed for, as was admitted among other factors, precisely that reason, especially in the formative stages.

Yet end user involvement is a sine qua non of good engineering practices (in the post-programmed card tabulator era, that is).

At what point will I be justified in saying that healthcare IT project leaders who skimp on clinician input are themselves eggheads, fools and gonifs?

-- SS

keith said...

Scott,

Your reading list if anything seems to make my point. There are alot of discussions from individuals in the IT and policy fileds within your list, but there is very little original research. Where are the measurements of quality before and after HIT implementation. Where are the studies looking for medication errors before and after implemnetation. Where are the analysis of what it costs to install and maintain these EHR packages?

Other than the study showing increased mortality following an installation at a pediatric hospital and a more recent study showing that quality measures are no better between paper charts and an electronic system, I have not seen these types of comparisons. Given hospital requirements for reporting to Medicare on quality measures and the fact we are in the midst of many hospitals converting to EMR, you would think someone would be looking at this.

Scot M Silverstein MD said...

Keith anonnymous at March 7, 2011 1:35:00 PM EST writes:

Your reading list if anything seems to make my point. There are alot of discussions from individuals in the IT and policy fields within your list, but there is very little original research.

I repeat:

The list in my view certainly does not lend support national rollouts of commercial health IT at this point in time.

As to some of my views on why there is a dearth of data on HIT risk, see a paper I wrote here entitled "Remediating an Unintended Consequence of Healthcare IT: A Dearth of Data on
Unintended Consequences of Healthcare IT."

-- SS

Anonymous said...

"As usual, the consumers (doctors) had virtually no input."

Yes, I've seen that argument to. However, this is a comment that seems to be made after the fact.

I've seen the other end of this: continual attempts to engage doctors in discussions before and during implementation fail because "I'm too busy ... I'm not a computer person ... I'm a clinician, not an IT person ...".
Excuses piled on excuses for not engaging with the project, and then complaining that they wheren't consulted.

Physicians lose their right to complain about consultation the moment they refuse to engage early in the process.

Scot M Silverstein MD said...

Anonymous March 7, 2011 4:28:00 PM EST writes:

Excuses piled on excuses for not engaging with the project, and then complaining that they wheren't consulted.

That's no my experience. For instance, I easily convinced half a dozen invasive cardiologists, enraged at two yeas of failed IT efforts in their facility, to meet with me several hours every week in redeveloping a system. They are among the busiest of clinicians.

I likewise attended their early morning (7 AM) business meetings.

Because I spoke the physicians' language, and was respectful of their responsibilities and obligations, they responded in kind.

There's a lesson there.

Physicians lose their right to complain about consultation the moment they refuse to engage early in the process.

They "lose their right to complain" the moment they refuse to engage early in the process?

The ethics, concern for patient safety, and management maturity of this position elude me.

-- SS

Anonymous said...

Anecdote and Medicine.

We are actually talking about two different things here.

1. Anecdotal reporting of a new and potentially exciting finding in Medicine is NEVER a reason to widely implement a new treatment or procedure. It represents the lowest category of evidence in any systematic review In any orthodox system of medicine in the developed world a new intervention would not be ratified or re-imbursed without EXTENSIVE study in Randomised trials - ie the mandatory three phase trial arrangement for new drugs.

2. Anecdotal reporting of side effects/failure for an implemented treatment is a crucial part of any risk management strategy within a healthcare setting. Individual incident reporting of harmful events AND near misses is crucial to help organisations (and regulatory agencies) understand where risks to patients and staff are to be found. A root cause analysis can then be undertaken and corrective measures introduced and their subsequent impact assessed. The process of 'closing the audit loop is required or no reduction in risk can be verified. This is separate from the regular audit cycle which each Department should apply to aspects of its work, usually reviewing a particular intervention in rotation.

In the above debate the pro e-Health lobby find themselves mis-interpreting both definitions. They support the positive anecdotes for the adoption of Electronic Health Records WITHOUT proper randomised evidence being available and they decry the anecdotes of negative experiences of implemented systems that in reality represent episodes of incident reporting. It appears the over-exuberant proponents of e-Health in general and for everything, need to attend revision courses in research methodology and risk management...

nb Irony of irony;

Victoria now has State-wide implementation of risk management software which for all healthcare staff is the obligatory reporting mechanism for all incidents and near-misses.

The system is so user unfriendly and time demanding virtually none of the busy hospital doctors I have spoken to access the system, even though 'training' has been undertaken. The system has been widely condemned at our Medical Staff Committee.

Victoria has recently congratulated itself for a fall in annual number of critical incidents occurring in public hospitals!

I will leave it to you to work out how such a positive risk management statistic could be generated in a healthcare system working near capacity with increasing year on year demand for its services...

InformaticsMD said...

Anonymous August 15, 2011 9:26:00 PM EDT said...

We are actually talking about two different things here.

1. Anecdotal reporting of a new and potentially exciting finding in Medicine ...

2. Anecdotal reporting of side effects/failure for an implemented treatment [incident reporting]

In the above debate the pro e-Health lobby find themselves mis-interpreting both definitions. They support the positive anecdotes for the adoption of Electronic Health Records WITHOUT proper randomised evidence being available and they decry the anecdotes of negative experiences of implemented systems that in reality represent episodes of incident reporting. It appears the over-exuberant proponents of e-Health in general and for everything, need to attend revision courses in research methodology and risk management...


This is brilliant. The loss of consideration of the two different types of "anecdotal reporting" is part of what I have termed the lack of the rigor of medicine itself in HIT.

-- SS

Anonymous said...

I agree with the earlier comments on Anecdote and Medicine. But the aparently clear-cut distinction between the usefulness of anecdotes is well, not all that clear cut. I just want to caution you that statements containing phrases such as "NEVER a reason", "the lowest category of evidence" and "systematic review", "any orthodox system of medicine" are bound to be wrong in quite a few situations where reason, clinical experience and a probably even a half a brain with an ounce of commonsense can figure the right answer. There is no need to be so absolute and require randomised trials for everything. Here, read for yourself another briliant argument why.

http://www.bmj.com/content/327/7429/1459.full