... It's the EMR "anecdotalists" (as opposed to the "Markopolists") who say that "anecdotes" of HIT-related injury are meaningless. They deem reports of safety issues and HIT-related misadventures and risk as simply "anecdotal", and that "anecdotes don't make evidence" (or "anecdotes don't make data").
For "anecdotes" of patient harm due to medical devices even from the most reliable of sources to be counted as "evidence" of device risk, apparently, the stories need to be blessed with Statistical Holy Water. The Holy Water must also be of a brand approved by the academic pundits.
For me, this is no longer merely a professional debate. My elderly relative became one of those "anecdotes" in May last year.
I address the casual, Dogbert-style, waving-of-the-hand "Bah!" dismissal of health IT harm "anecdotes" at numerous other posts as well, such as "EHR Problems? No, They're Merely Anecdotal" and "Health IT: On Anecdotalism and Totalitarianism".
Bah! Your Health IT adverse events reports are anecdotes, and anecdotes don't make data!In those posts I also mention how Australian informatics professor Dr. Jon Patrick had essentially hit the flaws of this argument out of the Southern hemisphere with a short editorial in the journal "Applied Clinical Informatics" entitled "The Validity of Personal Experiences in Evaluating HIT." That essay is free at the link and is worth reading.
Interestingly and thankfully, the "anecdotes are meaningless" crowd have now lost, and lost big - Supreme Court style. In fact, the U.S. Supreme Court has shown far more common sense than many esteemed academics and industry pundits.
As noted in this post at Derek Lowe's pharmaceutical industry "In the Pipeline" blog, the company that made "Zicam", a zinc-based over-the-counter cold remedy, tried to defend shareholder suits that the company withheld case reports of Zicam causing permanent loss of smell via arguing that such reports "did not reach a level of statistical significance", i.e., were "anecdotal." The case went to the U.S. Supreme Court.
The Supreme Court would have none of that argument:
"Matrixx’s [Zicam's manufacturer - ed.] premise that statistical significance is the only reliable indication of causation is flawed. Both medical experts and the Food and Drug Administration rely on evidence other than statistically significant data to establish an inference of causation. It thus stands to reason that reasonable investors would act on such evidence.
The full court decision is at this link: http://www.supremecourt.gov/opinions/10pdf/09-1156.pdf (PDF file), but a passage I consider key to this issue is as follows:
... We conclude that the materiality of adverse event reports cannot be reduced to a bright-line rule ... Because adverse reports can take many forms, assessing their materiality is a fact-specific inquiry, requiring consideration of their source, content, and context.
This is common sense incarnate. It applies not just to drugs, but to medical devices, to health IT, and to other domains as well.
In essence, it is saying that adverse events reports, especially repeated ones, from trustworthy sources are not to be lightly dismissed, but should serve at the very least as red flags that there may be a systemic problem requiring further investigation.
One wonders how and if public healthcare IT vendors will begin disclosing "anecdotal" reports of their products causing patient harm to their own stockholders.
One also wonders if the academic anecdotalists (up to the level of the chair of the Office of the National Coordinator for Health Information Technology at HHS) will cease their unfettered dismissal of health IT AE reports as mere "anecdotes" and therefore let's roll out this 100% beneficent technology nationwide ASAP:
"Nothing [ONC has] found would give them any pause that a policy of introducing EMR's could impede patient safety." - David Blumenthal
That sounds a bit like the refrain of the makers of Zicam.
One might also wonder if the anecdotalists merely lack common sense, or are using this form of epistemological dementia to obscure conflict of interest.
On a final note, my favorite comment at the aforementioned "In the Pipeline" blog story is this by anonymous commenter "Still Scared of Dinosaurs":
One of the most important ideas real statisticians must get into their heads is "Thou shalt not worship the 0.05 threshold". The whole concept of "statistical significance" for AEs is idiotic and the fact that Matrixx based any part of their defense on it indicates that their stupidity did not end when they named the company.
Perhaps this Dilbert cartoon is apropos to the Supreme Court decision:
-SS
Addendum:
I thought it appropriate to share these thoughts with the leadership of the Joint Commission, the organization that accredits healthcare organizations in the United States:
-- SSFrom: Scot Silverstein
Sent: Sunday, March 27, 2011 10:41 AM
To: MGiuntoli, Anita; Chassin, Mark; Schyve, Paul; Legaspi, Shirley
Cc: Ross Koppel; 'David Kreda'
Subject: Re: MATRIXX INITIATIVES, INC., ET AL. v. SIRACUSANO ET AL.Not a complaint this time [about health IT failure - ed.], but an observation.
The JC has noted health IT risks in the Sentinel Events Alert "Safely implementing health information and converging technologies" of 2008.
The company that made "Zicam", a zinc-based over-the-counter cold remedy, tried to defend shareholder suits that the company withheld case reports of Zicam causing permanent loss of smell via arguing that such reports "did not reach a level of statistical significance", i.e., were "anecdotal." The case went to the U.S. Supreme Court.
The Supreme Court would have none of that argument:
"Matrixx’s [Zicam's manufacturer] premise that statistical significance is the only reliable indication of causation is flawed. Both medical experts and the Food and Drug Administration rely on evidence other than statistically significant data to establish an inference of causation. It thus stands to reason that reasonable investors would act on such evidence.
The full court decision is at this link: http://www.supremecourt.gov/opinions/10pdf/09-1156.pdf (PDF file), but a passage I consider key to this issue is as follows:
... Because adverse reports can take many forms, assessing their materiality is a fact-specific inquiry, requiring consideration of their source, content, and context.
This is common sense incarnate. It applies not just to drugs, but to medical devices, to health IT, and to other domains as well.I believe JC should start to pay serious attention to "anecdotal reports" of health IT-caused patient injury, and consider reliable reporting of these events as an Accreditation standard.As I noted in my July 2009 JAMA letter to the editor "Health Care Information Technology, Hospital Responsibilities, and Joint Commission Standards" in response to Koppel and Kreda's JAMA article on HIT industry practices, "hold harmless" and "gag" clauses must go, and be replaced with proactive reporting of healthcare IT-related "events."Scot Silverstein
"But the supreme court is a bunch of conservative leaning bumblers who don't know nothing about medicine and doctoring or computin and health information."
ReplyDeleteWhich falacy is this?
My wife and I just returned from our yearly trip to France. This post reminded me that a couple of years ago I saw two articles: One outlined how all French students wanted to be philosophy majors. The second article highlighted how all French students felt the government owed them a job at a salary reflective of their university education.
ReplyDeleteThe end result is not hard to imagine. A large group of people who do not know how to do anything, telling those who do know, how wrong they are since the “data” does not reflect their view.
There is a growing impatience around the world with corrupt governments and government contracts. Anecdotal evidence does count since most people are aware of the philosophical nature of the discussions taking place in government regarding how work is suppose to be done. The same holds true for corporations.
There is a worldwide movement towards the right where people are held responsible for their actions. I can only hope this movement accelerates regarding medicine and medical technologies.
Medicine is ripe for the question: Does this work, and if not, who promoted the use of this product?
The answers are often only found in the resulting court cases.
Steve Lucas
"But the supreme court is a bunch of conservative leaning bumblers who don't know nothing about medicine and doctoring or computin and health information." Which falacy is this?
ReplyDeleteClassic ad hominem!
I guess it would not surprise me to hear that type of comment, but not made in jest, as a result of the court decision.
-- SS
Kudos to the Court for rendering a verdict against the corporation.
ReplyDeleteAs for HIT, the HIT vendors, their trade group HIMSS, and its legitimization pop-up $see$hit (CCHIT) knew all along about the defects, flaws,injuries and deaths from the equipment they were selling and falsely promoting as being safe on their websites.
Blumenthal knows also, of the dangers (from his own experience), and used the same depreciating expression to describe the adverse events, "anecdotes".
There needs to be an enterprising, young, and aggressive attorney to go after these deep pockets.
Anonymous March 27, 2011 3:05:00 PM ED writes:
ReplyDeleteBlumenthal ... used the same depreciating expression to describe the adverse events, "anecdotes".
I believe that is correct.
The next time doctors hear the word "anecdote" concerning their reports of HIT dangers, maybe they should shove some Zicam up the anecdotalist's nose.
-- SS
Another way of looking at this (“did not reach a level of statistical significance”) is that hypothesis testing is the wrong analysis method. Given that one has an adverse event reporting system, one should compute the rate of adverse events with its confidence interval (the confidence interval is especially important when the observed rate is zero). One can then decide if the adverse event rate is acceptable vs. the alternative of harm by not allowing the method (IT system, drug, etc.) to be used.
ReplyDeleteJan Krouwer said...
ReplyDeleteAnother way of looking at this (“did not reach a level of statistical significance”) is that hypothesis testing is the wrong analysis method.
One does not need an analysis method when pushing widgets in a profitable sector.
The only "hypothesis" in health IT seems to be that health IT is 100% beneficent, incapable of harm, therefore any reports of HIT-caused harm are simply erroneous.
See my posts on statements of HIT benefit certainty and beneficence, and lack of cause for concern about harm, by the chair of ONC and others.
-- SS