The Dangers of Critical Thinking in A Politicized, Irrational Culture

My early mentor in biomedicine Victor P. Satinsky MD lived by the credo "critical thinking always, or your patient's dead."

Unfortunately, the motto of today's degraded culture in biomedicine (and other domains) might well be "critical thinking, and your career is dead."

At "Health IT: On Anecdotalism and Totalitarianism" I posted these thoughts:

At the article Blumenthal on EMRs: Debate "raging" over competition vs. standards, ONC czar David Blumenthal is cited as saying several interesting things:

... EMRs make him a better physician, he said, recounting personal anecdotes of discovering patients' allergies through automated EMR alerts and using stored image date to more quickly get a diagnosis for a patient without subjecting them to more radiation and toxic radiation agents ...

It's the EMR "anecdotalists" (as opposed to the "Markopolists") who say that "anecdotes" of HIT-related injury are meaningless. They deem reports of safety issues and HIT-related misadventures and risk as simply "anecdotal", and that "anecdotes don't make evidence" (or "anecdotes don't make data").

Yet anecdotal reports of EMR "saves" are used by a czar to justify tens of billions of dollars of expenditures?

To the anecdotalists, I say: you can't have it both ways.

I also posted nearly the same complete Healthcare Renewal post to several mailing lists of the American Medical Informatics Association including the Clinical Information Systems working group (CIS-WG). CIS-WG is a mailing list read by something over 1000 healthcare informatics professionals at last time I had access to the statistics a few years ago.

I received some supportive replies from colleagues, including collaborators on the AHIMA (not AMIA) book we co-authored in 2009 entitled "H.I.T. or Miss: Lessons Learned from Health Information Technology Implementations" - itself not exactly a popular exercise among the strictly positivist informatics leadership class.

Now, I thought my posting on the double standard regarding "anecdotes" highly straightforward. From a high ranking academic leader of a major national informatics program, Bill Hersh at OHSU, however, the following reply was posted:

Scot,

For someone who is a faculty in informatics, I am surprised at how unfamiliar you are with the literature. There is solid evidence, much more than anecdotes, on the efficacy of health IT. Even Dr. Blumenthal himself has posted on that. (I think you are taking this quote out of context.

I am then served a platter of literature I must be "unfamiliar with" such as:

Goldzweig, C., Towfigh, A., et al. (2009). Costs and benefits of health information technology: new trends from the literature. Health Affairs, 28: w282-w293.

[Note - I had commented on and linked to this very article at this Aug. 29, 2010 post - ed.]

Garg, A., Adhikari, N., et al. (2005). Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. Journal of the American Medical Association, 293: 1223-1238.

Amarasingham, R., Plantinga, L., et al. (2009). Clinical information technologies and inpatient outcomes: a multiple hospital study. Archives of Internal Medicine, 169: 108-114.

Longhurst, C., Parast, L., et al. (2010). Decrease in hospital-wide mortality rate after implementation of a commercially sold computerized physician order entry system. Pediatrics, 126: 14-21.

Now, aside from the serious breach of academic etiquette of attacking the competence of your colleagues in a public forum, I seem to be hearing that it's OK to purvey positive anecdotes about health IT (usually based on weak retrospective observational studies alone, not randomized clinical trials), but not anecdotes of HIT malfunctions or of HIT-related adverse outcomes, since there's 'solid evidence' of the efficacy of health IT.

In plain English, an ad hominem fallacy is followed by an appeal to authority of sorts ("the literature") to justify public Pollyanna attitudes towards HIT by high ranking officials. And since the literature is so glowing, negative anecdotes must be of low worth.

[Jan. 2011 addendum - perhaps the literature's not so glowing - ed.]

Actually, the response in its entirety was a non sequitur to my post.

Others in cis-wg took affront. One of my book co-authors responded that:

I didn't read Scot's comment as saying that there is no data in support of EHRs .... despite a body of evidence, Dr. Blumenthal made a statement only about personal experience in what Scot quoted.

At the same time, ONC has asked for EHR users to share their positive experiences, but has not (as far as I have seen) asked for their failures. Quite frankly, the failures would be more instructive and would constitute a very valuable repository. ONC has also not shared the studies on the dangers and failures of EHR implementations with nearly the same passion as the successes. My point is that there is data and there is anecdote for both sides and ONC has not presented a balanced picture so that we can adequately address the real risks.

Another CIS-WG reader shared valuable observations:

Regarding Goldzweig:

Regarding studies conducted by the HIT leaders (e.g. Partners Vandy, Regenstrief, IHC, etc...): "Many of the new studies report modest or even no benefits of the new applications or changed functionalities."

Regarding studies of commercial HIT systems: "These study results were similar to those reported by the health IT leaders—most studies demonstrated modest benefits, some demonstrated no benefits, and a few demonstrated marked benefits."

Regarding Adhikari:

The CDSS improved practitioner performance in 62 (64%) of the 97 studies assessing this outcome,

52 trials assessed 1 or more patient outcomes, of which 7 trials (13%) reported improvements.

And so on.

In other words, the literature's mixed.

Finally, the knock-the-ball-out-of-the-park response came from a Medical Informatics researcher Down Under:

I think such defences are particularly unuseful especially with respect to the dismissal of personal stories and experiences as "anecdotes", hence committing them to the realm of folklore. I offer these notions as a counterpoint.

Discounting Anecdotes:

1. Is a perfidious and specious act.

2. It denies early warning signs of problems.

3. It denies a voice and disempowers the working clinical community who have to operationalise decisions made by others.

4. It denies a route to process improvement within an institution - which is most important for EBM and incremental review of local processes.

5. It defends software manufacturers from fault rectification - cuts off even a need to deliberate on it. Critics of the value of anecdotes are squarely on the side of the faulty and deficient manufacturer.

6. A rule of project management is that projects consist of 3 components, cost, quality and time and if their needs to be a compromise it has to be on quality. Anecdotes are early warning signs of such a compromise.

I, of course, added that ignoring "anecdotes" of HIT problems was even more cavalier if one recognized the context of the stories, that is, that they arise in an environment hostile to diffusion through contractual arrangements, poorly recognized reporting resources, fear, etc. Understood in context, they should be receiving more research attention than otherwise, and certainly not ignored.

However, the comment about my purported lack of knowledge of the literature was sent out to 1000+ people by a nationally-recognized informatics leader, people who may or may not read the followup in detail.

This is unfortunate and perhaps reflects the ethos of our day.

-- SS

Addendum:

Also see the Aug. 2011 post "From a Senior Clinician Down Under: Anecdotes and Medicine, We are Actually Talking About Two Different Things" for a truly stunning takedown of the "anecdotes" canard, which amounts to conflating risk management with scientific discovery.