Wednesday, August 17, 2011

From a Senior Clinician Down Under: Anecdotes and Medicine, We are Actually Talking About Two Different Things

A poster who wishes to remain anonymous, a Senior Clinician in the state of Victoria, Australia, added this comment to my March 2011 post on 'anecdotes.' (That post was entitled "Australian ED EHR Study: An End to the Line "Your Evidence Is Anecdotal, Thus Worthless?".)
He makes a critical point I think has gotten lost in the HIT domain (emphases mine):

August 15, 2011 9:26:00 PM EDT said...

Anecdote and Medicine.

We are actually talking about two different things here.

1. Anecdotal reporting of a new and potentially exciting finding in Medicine is NEVER a reason to widely implement a new treatment or procedure. It represents the lowest category of evidence in any systematic review In any orthodox system of medicine in the developed world a new intervention would not be ratified or re-imbursed without EXTENSIVE study in Randomised trials - ie the mandatory three phase trial arrangement for new drugs.

2. Anecdotal reporting of side effects/failure for an implemented treatment is a crucial part of any risk management strategy within a healthcare setting. Individual incident reporting of harmful events AND near misses is crucial to help organisations (and regulatory agencies) understand where risks to patients and staff are to be found. A root cause analysis can then be undertaken and corrective measures introduced and their subsequent impact assessed. The process of 'closing the audit loop' is required or no reduction in risk can be verified. This is separate from the regular audit cycle which each Department should apply to aspects of its work, usually reviewing a particular intervention in rotation.

In the above debate the pro e-Health lobby find themselves mis-interpreting both definitions.

They support the positive anecdotes for the adoption of Electronic Health Records WITHOUT proper randomised evidence being available and they decry the anecdotes of negative experiences of implemented systems that in reality represent episodes of incident reporting.

It appears the over-exuberant proponents of e-Health in general and for everything, need to attend revision courses in research methodology and risk management... [a fascinating observation - ed.]

nb Irony of irony;

Victoria now has State-wide implementation of risk management software which for all healthcare staff is the obligatory reporting mechanism for all incidents and near-misses.

The system is so user unfriendly and time demanding virtually none of the busy hospital doctors I have spoken to access the system, even though 'training' has been undertaken. The system has been widely condemned at our Medical Staff Committee.

Victoria has recently congratulated itself for a fall in annual number of critical incidents occurring in public hospitals!

I will leave it to you to work out how such a positive risk management statistic could be generated in a healthcare system working near capacity with increasing year on year demand for its services...

Per these observations:

A critical distinction that seems to have become lost, even among the Medical Informatics academic elite (see, for instance, my Sept. 2010 post "The Dangers of Critical Thinking in A Politicized, Irrational Culture"), is the distinction between research observations on the one hand, and risk management-relevant incident reports on the other.

It seems a form of erroneous thinking or logical fallacy.

The lost distinction between research methods and risk management methods, that require very essential, very different consideration of "anecdotes", and the conflation of the two types of "anecdotes", are brilliant observations.

Finally, the loss of consideration of the distinctions between the two different types of "anecdotal reporting" is part of what I have termed the lack of the rigor of medicine itself in HIT.

-- SS


Jon Patrick said...

The distinction between methodologies of risk management and medical science is a vitally important distinction I have not seen previously so clearly enunciated as presented by this author. It constitutes a resounding complementation to my editorial on personal experience published in the ACI earlier this year
Whilst many writers have pointed out that the quality of HIT has to be determined by among other things its RISK, it may well be that the ultimate generalisation is that all aspects of HIT point back to risk, despite all the arguments that it points to increased productivity. Would it not be true that the increases in productivity cannot be viewed as any isolated characteristic but must be traded against the increased risk they create whether it be to patients' or medical staff well-being.

Anonymous said...

As individual cases, the deaths from EMR and CPOE systems were referred to as anecdotes by former ONC Director, David Blumenthal.

When an analysis of the care of the dead patients is performed, there are connecting threads. ie patterns to the deaths. Very simply, the patients died from neglect, abject neglect.

The complex user unfriendly EMRs and CPOEs were the object of the cognition and attention of the care team. The real patients suffered while the care team clicked. Key medications and/or clinical findings and transitions in clinical status were missed or recognized too late to successfully treat.

InformaticsMD said...

Jon Patrick said...

Whilst many writers have pointed out that the quality of HIT has to be determined by among other things its RISK, it may well be that the ultimate generalisation is that all aspects of HIT point back to risk, despite all the arguments that it points to increased productivity.


I knew you would find this post of interest.

My mother would heartily agree with you, were she still alive.


Anonymous said...

The evidence showing benefit for HIT in medical care is anecdotal and evangelical.

The epidemiological studies published to date do not demonstrate any benfit in costs or outcomes.

The US government is wasting $ billions, that will go to $ trillions when all expenditures for wiring doctors and hospitals (eg attendant maintenance, employee, and loss of productivity costs)are factored in.

Anonymous said...

I would like to thank Jon Patrick for his flattering comments. I have read his editorial at ACI and it should be prescribed reading for all Health bureaucrats, Politicians, Hospital CEOs and CIO's and Boards.

I have also recently come across the work of Barbara Tuchman and her reflections on Government policy persistently pursued against their own real interests. Her definition of Folly was widely applied to historical foreign policy fiascos, but I think you will see how it could be relevant to NPfIT, HiTECH, HealthSmart and other ill considered e-Health initiatives in the English speaking world;

"In her book The March of Folly, US historian Barbara Tuchman cited the Trojan Horse story as showing that humankind in the shape of the citizens of Troy "is addicted to pursuing policy contrary to self-interest". For Tuchman, the fall of Troy was only the first of many great acts of folly in history.

Describing folly as "a perverse persistence in a policy that is demonstrably unworkable or counter-productive", she says a policy can be identified as folly if it meets three tests.

It must have been perceived as counter-productive at the time and not just by hindsight; a feasible alternative must have been available; and the policy must be that of a group and persist over a span of time, not the act of an individual ruler."

A suitable title for a book?
"Health IT in Australia, the reformers Phar Lap or Trojan Horse?"

Anonymous said...

I spoke to a young man who voted for hope and change last time. Now he says he doesn't see why he should pay taxes just so that money can be spent by the NIH forcing researchers to spend 75% of their time on grant writing leaving less than 25% for actual research.

He was disgusted with the way the system talks about all these high ideals yet when push comes to shove it is always who's butt you are licking and the self interests involved.

It’s a regal class where the emperors wear no cloths and we all say how well they dress and eat their scraps.

The number of optimists becoming cynics is as large as I have ever seen.

Totally predictable of course.

Anonymous said...

Off the record, Aussie government Health IT people will tell you the projects there are a disaster with vendors running amuck. One mid level manager just told me that yesterday. S/He'd never say that of course on the record because s/he knows he'd be let go at the next disconnected opportunity or flatlined. Force fitting generalized Cerner into the Eye and Ear specialty was a particular example.