Those Who Dismiss Healthcare (and Healthcare IT) Adverse Events Reports as Mere "Anecdotes" Have Lost - Supreme Court-Style

At my Sept. 2010 post "The Dangers of Critical Thinking in A Politicized, Irrational Culture" I wrote:

... It's the EMR "anecdotalists" (as opposed to the "Markopolists") who say that "anecdotes" of HIT-related injury are meaningless. They deem reports of safety issues and HIT-related misadventures and risk as simply "anecdotal", and that "anecdotes don't make evidence" (or "anecdotes don't make data").

For "anecdotes" of patient harm due to medical devices even from the most reliable of sources to be counted as "evidence" of device risk, apparently, the stories need to be blessed with Statistical Holy Water. The Holy Water must also be of a brand approved by the academic pundits.

For me, this is no longer merely a professional debate. My elderly relative became one of those "anecdotes" in May last year.

I address the casual, Dogbert-style, waving-of-the-hand "Bah!" dismissal of health IT harm "anecdotes" at numerous other posts as well, such as "EHR Problems? No, They're Merely Anecdotal" and "Health IT: On Anecdotalism and Totalitarianism".

Bah! Your Health IT adverse events reports are anecdotes, and anecdotes don't make data!

In those posts I also mention how Australian informatics professor Dr. Jon Patrick had essentially hit the flaws of this argument out of the Southern hemisphere with a short editorial in the journal "Applied Clinical Informatics" entitled "The Validity of Personal Experiences in Evaluating HIT." That essay is free at the link and is worth reading.

Interestingly and thankfully, the "anecdotes are meaningless" crowd have now lost, and lost big - Supreme Court style. In fact, the U.S. Supreme Court has shown far more common sense than many esteemed academics and industry pundits.

As noted in this post at Derek Lowe's pharmaceutical industry "In the Pipeline" blog, the company that made "Zicam", a zinc-based over-the-counter cold remedy, tried to defend shareholder suits that the company withheld case reports of Zicam causing permanent loss of smell via arguing that such reports "did not reach a level of statistical significance", i.e., were "anecdotal." The case went to the U.S. Supreme Court.

The Supreme Court would have none of that argument:

"Matrixx’s [Zicam's manufacturer - ed.] premise that statistical significance is the only reliable indication of causation is flawed. Both medical experts and the Food and Drug Administration rely on evidence other than statistically significant data to establish an inference of causation. It thus stands to reason that reasonable investors would act on such evidence.

The full court decision is at this link: http://www.supremecourt.gov/opinions/10pdf/09-1156.pdf (PDF file), but a passage I consider key to this issue is as follows:

... We conclude that the materiality of adverse event reports cannot be reduced to a bright-line rule ... Because adverse reports can take many forms, assessing their materiality is a fact-specific inquiry, requiring consideration of their source, content, and context.

This is common sense incarnate. It applies not just to drugs, but to medical devices, to health IT, and to other domains as well.

In essence, it is saying that adverse events reports, especially repeated ones, from trustworthy sources are not to be lightly dismissed, but should serve at the very least as red flags that there may be a systemic problem requiring further investigation.

One wonders how and if public healthcare IT vendors will begin disclosing "anecdotal" reports of their products causing patient harm to their own stockholders.

One also wonders if the academic anecdotalists (up to the level of the chair of the Office of the National Coordinator for Health Information Technology at HHS) will cease their unfettered dismissal of health IT AE reports as mere "anecdotes" and therefore let's roll out this 100% beneficent technology nationwide ASAP:

"Nothing [ONC has] found would give them any pause that a policy of introducing EMR's could impede patient safety." - David Blumenthal

That sounds a bit like the refrain of the makers of Zicam.

One might also wonder if the anecdotalists merely lack common sense, or are using this form of epistemological dementia to obscure conflict of interest.

On a final note, my favorite comment at the aforementioned "In the Pipeline" blog story is this by anonymous commenter "Still Scared of Dinosaurs":

One of the most important ideas real statisticians must get into their heads is "Thou shalt not worship the 0.05 threshold". The whole concept of "statistical significance" for AEs is idiotic and the fact that Matrixx based any part of their defense on it indicates that their stupidity did not end when they named the company.

Perhaps this Dilbert cartoon is apropos to the Supreme Court decision:


-SS

Addendum:

I thought it appropriate to share these thoughts with the leadership of the Joint Commission, the organization that accredits healthcare organizations in the United States:

From: Scot Silverstein
Sent: Sunday, March 27, 2011 10:41 AM
To: MGiuntoli, Anita; Chassin, Mark; Schyve, Paul; Legaspi, Shirley
Cc: Ross Koppel; 'David Kreda'
Subject: Re: MATRIXX INITIATIVES, INC., ET AL. v. SIRACUSANO ET AL.

Not a complaint this time [about health IT failure - ed.], but an observation.

The JC has noted health IT risks in the Sentinel Events Alert "Safely implementing health information and converging technologies" of 2008.

The company that made "Zicam", a zinc-based over-the-counter cold remedy, tried to defend shareholder suits that the company withheld case reports of Zicam causing permanent loss of smell via arguing that such reports "did not reach a level of statistical significance", i.e., were "anecdotal." The case went to the U.S. Supreme Court.

The Supreme Court would have none of that argument:

"Matrixx’s [Zicam's manufacturer] premise that statistical significance is the only reliable indication of causation is flawed. Both medical experts and the Food and Drug Administration rely on evidence other than statistically significant data to establish an inference of causation. It thus stands to reason that reasonable investors would act on such evidence.

The full court decision is at this link: http://www.supremecourt.gov/opinions/10pdf/09-1156.pdf (PDF file), but a passage I consider key to this issue is as follows:

... Because adverse reports can take many forms, assessing their materiality is a fact-specific inquiry, requiring consideration of their source, content, and context.

This is common sense incarnate. It applies not just to drugs, but to medical devices, to health IT, and to other domains as well.

I believe JC should start to pay serious attention to "anecdotal reports" of health IT-caused patient injury, and consider reliable reporting of these events as an Accreditation standard.

As I noted in my July 2009 JAMA letter to the editor "Health Care Information Technology, Hospital Responsibilities, and Joint Commission Standards" in response to Koppel and Kreda's JAMA article on HIT industry practices, "hold harmless" and "gag" clauses must go, and be replaced with proactive reporting of healthcare IT-related "events."

Scot Silverstein

-- SS