Ioannidis et al.: What Happens When Underperforming Big Ideas in Research [such as Healthcare IT Exceptionalism] Become Entrenched?

Some years ago, John P. A. Ioannidis, MD wrote this piece:

"Why Most Published Research Findings Are False", John P. A. Ioannidis, PLoS medicine, 2005 August; 2(8): e124

He wrote:

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.

In other words, in the all-too-common insufficiently powered studies, and even seemingly robust studies in domains with small effect sizes, financial interests, prejudices and other factors more often than not produce false results.

Ioannidis and co-authors recently took their sword to "underperforming Big Ideas in research" (including the "miracles" touted by hyper-enthusiasts such as in genomics and in cybernetics), via a new JAMA viewpoint piece:

What Happens When Underperforming Big Ideas in Research Become Entrenched?
Michael J. Joyner, MD1; Nigel Paneth, MD, MPH2; John P. A. Ioannidis, MD, DSc3
JAMA. Published online July 28, 2016. doi:10.1001/jama.2016.11076
http://jama.jamanetwork.com/article.aspx?articleid=2541515

For several decades now the biomedical research community has pursued a narrative positing that a combination of ever-deeper knowledge of subcellular biology, especially genetics, coupled with information technology will lead to transformative improvements in health care and human health. In this Viewpoint, we provide evidence for the extraordinary dominance of this narrative in biomedical funding and journal publications; discuss several prominent themes embedded in the narrative to show that this approach has largely failed; and propose a wholesale reevaluation of the way forward in biomedical research.

The key word is "narrative."   As per Hayek, those with little real-world operational experience, i.e., intellectuals and academics, often the uncritical cheerleaders for electronic records despite considerable downsides, have only the "narrative" upon which they base their beliefs in healthcare IT exceptionalism:

It is perhaps the most characteristic feature of the intellectual that he judges new ideas not by their specific merits but by the readiness with which they fit into his general conceptions, into the picture of the world which he regards as modern or advanced. . . . As he knows little about particular issues, his criterion must be consistency with his other views and suitability for combining into a coherent picture of the world. . . . It is the intellectuals in this sense who decide what views and opinions are to reach us, which facts are important enough to be told to us, and in what form and from what angle they are to be presented. Whether we shall ever learn of the results of the work of the expert and the original thinker depends mainly on their decision.

(I can add that blogs have to some small degree ameloriated "whether we shall ever learn of the results of the work of the expert and the original thinker", but only to a small degree.)

The "general conception" in cybernetics is that computers are a silver bullet in any domain, and can only result in massive improvements. 

My experience for the past twenty+ years in the Electronic Medical Records/clinical information technology domain, where quality, safety, usability, confidentiality, and other critical real-world issues have been ignored in favor of EHR hyper-enthusiasm, supports Hayek's observations regarding prevalent unfettered beliefs in healthcare IT exceptionalism.

Ioannidis et al. state the factual situation with EHR technology unapologetically, clearly and succinctly:

... The financial and clinical benefits predicted from shifting to EHRs have also largely failed to materialize because of difficulties in interoperability, poor quality, and accuracy of the collected information; cost overruns associated with installation and operation of EHRs at many institutions; and ongoing privacy and security concerns that further increase operational costs.

I would change "interoperability" to "operability."  Otherwise, they're quite correct.  For example, the "Big Data" hyper-enthusiasts quite irrationally believe data from these systems - as they are today -  will somehow "revolutionize" medicine, while at the very same time the IT industry itself and its pundits ignore fundamental precepts of computer science, information science, biomedical informatics, biomedicine and biomedical research itself. 

Some of the hyper-enthusiasts have made predictions that are astonishingly naive, delusionally grandiose and just plain perverse, e.g., see for instance my Jan. 2014 post "Computers + a few docs can manage 'an entire city', and other cybernetic miracles" at http://hcrenewal.blogspot.com/2014/01/computers-few-docs-can-manage-entire.html . 

The new JAMA paper continues:

... These features make the use of EHRs for research into the origins of disease, as proposed in the Precision Medicine Initiative, highly problematic.No clearly specified targets for either improved outcomes or reduced costs have been developed to assess the performance efficiency of EHRs.

Those targets were never specified, but The Market seems to have corrected for that, e.g., via this Jan. 2015 letter from ~40 different medical societies:

 Full letter to HHS available at http://mb.cision.com/Public/373/9710840/9053557230dbb768.pdf

The authors continue:

... Although it is difficult to argue for a return to paper records, any claim of future transformation of the medical record should include well-defined accountability and review mechanisms. Otherwise, the health care system may become hostage, wasting increasing resources to continuously upgrade electronic technology without really helping patients.

It is clear to me that the health care system and its clinicians are already hostage to the cybernetic hyper-enthusiasts, as evidenced by letters such as the above and many other sources about the mayhem being caused, e.g., a small sampling on this blog at query links http://hcrenewal.blogspot.com/search/label/glitch, http://hcrenewal.blogspot.com/search/label/healthcare%20IT%20difficulties and http://hcrenewal.blogspot.com/search/label/medical%20record%20confidentiality.

Finally, I disagree with the authors that "it is difficult to argue for a return to paper records."

Paper has its proper place, and "paperless" is a utopian dream of the hyper-enthusiasts that causes significant damage to the primary role of clinicians - to take care of patients.  I make this argument (with a real-world, highly successful example of my own creation) at my Aug. 9, 2016 post "More on uncoupling clinicians from EHR clerical oppression" at http://hcrenewal.blogspot.com/2016/08/more-on-uncoupling-clinicians-from-ehr_91.html.

In summary, the authors of this JAMA piece clearly and succinctly break through the "narrative" about hyper-enthusiast dominated fields, including clinical information technology and the belief in 
healthcare IT exceptionalism.
 
Sadly, theirs is almost a single voice in a wilderness dominated by the hyper-enthusiasts - and the profiteers.

-- SS

Two recent stories of EHR un-exceptionalism, and a connection to prior HC Renewal posts including my own experiences

The following stories recently appeared in the press about the "un-exceptionalism" (or, I might more accurately state, the negative exceptionalism) of today's EHR technology.

Regarding each organization mentioned, I have had personal experience that, in retrospect, aligns with views I expressed about the organizations years ago.

First:

EHR safety goes to court
By Lisa Schencker

June 25, 2016

http://www.modernhealthcare.com/article/20160625/MAGAZINE/306259982

One patient's blood pressure plummeted dangerously after he was allegedly discharged with the wrong medications. In another instance, a physician couldn't place a pharmacy order for a newborn to receive vitamin K, which is given to babies to prevent serious bleeding.

On several other occasions, patients weren't accurately tracked, creating potential problems getting drugs to them.

Each of these alleged mishaps occurred at PinnacleHealth, a three-hospital system based in Harrisburg, Pa. PinnacleHealth blames each of the mishaps on its electronic health records vendor, Siemens; Cerner Corp. purchased Siemens' health IT business in February 2015.

The relationship between healthcare system and vendor devolved into this dueling lawsuit:

The incidents came to light as part of a breach-of-contract lawsuit Cerner filed against PinnacleHealth last year after the system, which had used Siemens as a vendor for 20 years, sharply curtailed its relationship and entered into a contract with a competing EHR vendor, Epic Systems Corp. PinnacleHealth related the incidents in its counterclaim; the counterclaim was filed in March of this year in state court in Pennsylvania, where it is seeking damages for Cerner's alleged fraud and breach of contract.

(I am cited in the article with respect to hold harmless clause-related issues not relevant to this blog posting.)

Cerner Corp. purchased Siemens' HIT business in Feb. 2015.  Prior to that, Siemens was an independent vendor of an EHR suite known as Soarian, deployed at PinnacleHealth as early as 2008.

See for instance http://news.usa.siemens.biz/press-release/customer-solutions/soarian-plan-care-assists-clinicians-developing-and-deploying-goal-

Soarian Plan of Care Assists Clinicians in Developing and Deploying Goal-Oriented Plans of Care
Monday, February 25, 2008

Siemens (http://www.usa.siemens.com/healthcareit) today announced that PinnacleHealth, a 750-bed, non-profit hospital and healthcare system serving Central Pennsylvania, and CentraState Healthcare System, a 272-bed, non-profit medical facility located in Freehold, N.J., have each signed on to serve as beta sites for Soarian® Plan of Care, a solution designed to support inter-disciplinary care teams in defining and helping to manage patient problems, interventions and expected outcomes, to provide care in order to achieve desired outcomes and to generate new knowledge at the point of care.

Notwithstanding my warnings on this blog and elsewhere that hospitals should NEVER be "beta testing sites" for experimental HIT software used on live patients without informed consent, that the relationship devolved to dueling lawsuits involving safety is not a surprise to me.

Relating to my own experiences as posted previously on this blog:

On August 20, 2009 I posted "Why Siemens Healthcare Fails" (http://hcrenewal.blogspot.com/2009/08/why-siemens-healthcare-fails.html).  In that post I noted regarding a job posting by Siemens for a "physician consultant" on HIT implementation:

... This [Siemens] job description might better be described as "glorified salesperson." It might be a good exit route for a "techie doc" (usually, someone who knows just enough about HIT to be destructive) who hates the current practice environment. It might also be good for managers who don't want knowledgeable experts pointing out their bad decisions and mismanagement, but I think a global company like Siemens should be setting its sights higher in such a crucial area as electronic medical records and clinical IT consultants.

I would not want such a physician advising or supporting complex HIT projects at my organization.

I also recalled my own Siemens face time:

... I spent time at Siemens Healthcare headquarters in Erlangen in 2000, and was offered gracious hospitality and a position overseeing the Soarian cardiology suite [as a result of my work in development of an extensive cardiology information system as CMIO at Christiana Care in Delaware from 1996-8]. The people I met in Erlangen then seemed extremely competent and informatics-savvy, but I turned the offer down through no fault of Siemens. I'd received a near-simultaneous offer (FAXed to my hotel in Erlangen, in fact) from pharma that involved a much stronger management role.

I understand through conversations over the past few years with current and ex-Siemens personnel that most of the Siemens personnel I'd met in Germany in 2000 are no longer with the company. I was told they'd performed suboptimally after the acquisition/merger with Shared Medical Systems (SMS) in Malvern, PA. (I do not find that credible, and would find it far easier to accept that the problems were on the American side, but that is a personal opinion.)

I'd informed my German hosts in 2000 that they'd better be very, very careful about acquisition of an American HIT company due to my experiences with a number of such companies, which I found highly political and highly ignorant of EHR quality issues.

Then, in my Feb. 2016 post "Plaintiff's Lawyers Are The Cause of EHR Problems?" (http://hcrenewal.blogspot.com/2016/02/himss-2016-presentation-plaintiffs.html) I recounted further experience with Siemens:

... an anecdote regarding the health IT Industry in the U.S.: the cardiology information system I developed, linked to above in the 2nd bullet point, was seen in 2000 by German engineers at Siemens Healthcare Erlangen as exemplary, and they offered me a position to further develop it, that I declined due to a simultaneous superior offer from Merck Research Labs.  However, in 2007 when I again spoke to Siemens, this time to Americans at the former Shared Medical Systems in Malvern, PA that had been acquired by Siemens, they found a system that actually produced clear, detailed outputs in a critical care area and was in use at the time in a major healthcare system in the region "impractical" - and never followed up with me.  Pearls before....)

 Finally, Siemens seems to have ignored a well-qualified informatics internal expert on explicit EHR safety warnings, as in my Feb. 8, 2010 post "A Lawsuit Over Healthcare IT Whistleblowing and Wrongful Discharge: Malin v. Siemens Healthcare" (http://hcrenewal.blogspot.com/2010/02/lawsuit-over-health-it-whistleblowing.html):

• Siemens Medical's physician internal consultants (skilled anesthesiologists/informatics specialists), apparently in writing and including a remediation plan, opined that an IT system developed for critical care had numerous severe flaws, of sufficient seriousness that in their opinion the systems could harm or even kill patients if deployed;

• These were warnings of defects and flaws in IT in the worst possible medical environment, critical care with the sickest and most vulnerable of patients, not some ambulatory clinic or primary care office;

• Allegations were raised that the warnings were ignored, with at least one of the physicians, Dr. Malin, who was strongly concerned and vocal about the risks then being told his position was being eliminated. [It is not a surprise to note this rings alarm bells about the possibility of wrongful discharge based on retaliation and/or "greasing the skids" to eliminate potential whistleblowers or "non-team players" who could delay release of the software and affect revenue - ed.];

• The company possibly ignored the remediation plans of their own clinician/informatics experts;

• There were allegations of company misrepresentations about the new system to the FDA;

• There were allegations of decision making on these issues by non-clinician IT managers lacking healthcare or healthcare informatics expertise.

• A wrongful discharge complaint and then lawsuit were filed by Dr. Malin on the basis of violation of the whistleblower protection provisions of the Sarbanes-Oxley Act of 2002 (SOX), 18 U.S.C. § 1514A ("Civil action to protect against retaliation in fraud cases - Whistleblower Protection for Employees of Publicly Traded Companies");

• The suit was dismissed on the apparent technicality that Siemens Medical in the U.S. is not subject to the provision in the SOX Act as is the publicly-traded corporate parent, Siemens AG; from document #4, pg 15:

... Health Services is a wholly-owned subsidiary of Siemens Med ... Siemens Med is a wholly-owned subsidiary of Siemens Corporation, which is indirectly owned, through two intervening layers, by Siemens AG, a German company that is publicly traded as defined by § 1514A of SOX ... Of these entities, only Siemens AG is a publicly-traded company. While both Health Services and Siemens Med are incorporated in Delaware and located in Malvern, Pennsylvania, they are separately incorporated entities.

• Siemens Medical in their response to the suit denied the most severe allegations regarding the IT defects, but this issue was not followed up upon due to the lawsuit's dismissal on SOX issues despite the obvious potential public hazards the allegations of IT defects could represent;

• In fact a US District Judge in the case, Peter J. Messitte, in the period after allowing Malin to prove the validity of SOX towards his case, opined that "No other discovery will be permitted, including but not limited to the alleged safety problems of Defendants’ product", document 4 above, Judge Messitte opinion, US District Court Maryland, pg. 25. [While perhaps understandable from a legal perspective, injured or dead patients don't really care about what legal precedents got them into the injured or dead state - ed.]

• Siemens Medical admitted in their responses to the suit that some of the software in question, actually put in use in hospitals, was in fact "beta" software, i.e., experimental (per Item 38 in 'Siemens Answer to Complaint' document).

There's that "beta-software" issue again.

So, the acquirer (Cerner) of a product of an American Siemens HIT subsidiary that appeared to have what I consider defective HIT talent management practices (as exemplified by the aforementioned JD); bizarre views about successful, high-quality EHR software (seen by their German counterparts a few years prior as exceptional); and that appears to have not been very happy about its own employee warning explicitly of beta ICU software being so bad it could kill patients is now embroiled in litigation regarding safety by a former customer.

None of this is surprising to me at all.  Incompetence can only go so far in healthcare before things blow up.

An unanswered question, of course, is whether patients were harmed or killed as a result of Siemens software, including its beta tests on unsuspecting patients.

---------------------------------

Here is a second story of EHR un-exceptionalism with a personal twist, this article in the Journal of the American Medical Informatics Association (JAMIA):

Comparison of accuracy of physical examination findings in initial progress notes between paper charts and a newly implemented electronic health record, June 28, 2016
http://jamia.oxfordjournals.org/content/early/2016/06/28/jamia.ocw067

Introduction There have been several concerns about the quality of documentation in electronic health records (EHRs) when compared to paper charts. This study compares the accuracy of physical examination findings documentation between the two in initial progress notes.

Methodology Initial progress notes from patients with 5 specific diagnoses with invariable physical findings admitted to Beaumont Hospital, Royal Oak [Michigan, near Detroit -ed.], between August 2011 and July 2013 were randomly selected for this study. A total of 500 progress notes were retrospectively reviewed. The paper chart arm consisted of progress notes completed prior to the transition to an EHR on July 1, 2012. The remaining charts were placed in the EHR arm. The primary endpoints were accuracy, inaccuracy, and omission of information. Secondary endpoints were time of initiation of progress note, word count, number of systems documented, and accuracy based on level of training.

Results The rate of inaccurate documentation was significantly higher in the EHRs compared to the paper charts (24.4% vs 4.4%). However, expected physical examination findings were more likely to be omitted in the paper notes compared to EHRs (41.2% vs 17.6%). Resident physicians had a smaller number of inaccuracies (5.3% vs 17.3%) and omissions (16.8% vs 33.9%) compared to attending physicians.

Conclusions During the initial phase of implementation of an EHR, inaccuracies were more common in progress notes in the EHR compared to the paper charts. Residents had a lower rate of inaccuracies and omissions compared to attending physicians. Further research is needed to identify training methods and incentives that can reduce inaccuracies in EHRs during initial implementation.

Apparently not covered in the article was the issue of patient harms that might have occurred (and could still occur) due to the "significantly higher rate of inaccurate documentation" in the EHR.   

"Beaumont Hospital" caught my eye, as I had interviewed for the CMIO role there in 2007 or early 2008.  My experiences are memorialized in an anonymized post at my Drexel informatics site which I reproduce in part here (the full essay is at http://cci.drexel.edu/faculty/ssilverstein/cases/?loc=cases&sloc=an_expert_says):

Sure, the experts think you shouldn't ride a bicycle into the eye of a hurricane, but we have our own theory

A medical informaticist who formerly held a “Director of Informatics” role a number of years ago in a very large hospital system, and who left the role due to a toxic management environment and lack of authority commensurate with responsibility, was seeking applied Chief Medical Informatics Officer (CMIO) positions once again.  ... He makes the following observations after completing two full rounds of interviews at a prestigious hospital system similar in size to his erstwhile employer, in a very competitive environment, that recently experienced a decline in its clinical quality stats. The organization feels the quality stats themselves were inaccurate, in part due to lack of good healthcare IT.

From what the informaticist was able to gather, their leadership was displeased. Board members were seasoned executives from a heavy-manufacturing industry that is extremely dependent on information technology and concurrent supply chain data. These executives apparently recommended that the organization move quickly on implementing EHRs.

The organization is thus planning to implement EHRs for thousands of physicians, most of whom are not employed by the hospital but are independent private practitioners, and likely to be skeptical or concerned about time impact and “grading” that could affect their livelihoods.  The hospital leaders also wanted to create integrated systems drawing on EHR data to automate quality reporting to regulatory agencies, as well as to support ongoing, funded clinical drug and device trials.

The informaticist was interviewed by the usual mix of clinician eager adopters, clinician skeptics, knowledgeable executives, skeptical executives who knew little about clinical IT, IT personnel who seemed overconfident given the enormity of the tasks at hand, and those who were clearly frightened by the prospect of being held accountable for a project of this magnitude. In the end, the informaticist did not get the position due to the organizational leaders being adamant the incumbent CMIO needed to also practice medicine.  The informaticist had explained on the first round that he believed a leadership role in a project of such magnitude and challenge called for the highest levels of executive presence and freedom from distraction, thus he did not intend to practice medicine (he had not practiced in his former Director of Informatics role for the same reasons).

He’d thought this issue had been settled after the first round of interviews, leading to the invitation for round two. This line of questioning was revisited, however, in round two in a group interview setting. The group interview was attended by a number of people with whom he’d already discussed this issue via individual meetings in round one. This suggested his time was being wasted and was rather annoying, especially considering that the informaticist had flown cross-country not once but twice to an organization not in consensus about a very basic hiring requirement ... It is not as if the organization had a doctor shortage, or that such a role would have ample free time where the incumbent would be idly sitting at their desk unless this time was absorbed seeing patients in the clinic. 

The stated reason for the organization’s wanting the CMIO to practice medicine was “to have credibility with the doctors.”  The informaticist explained that he’d found this not to be the case, that physicians being put “under the gun” of using EHR’s were generally more concerned that the CMIO had the executive authority to best represent their interests and understood medicine from training and practice at some point, not necessarily concurrently with the CMIO role.  Interestingly, there is no empirical research on this point, so the issue was the informaticist’s experience vs. the hospital’s ‘second-guessing’ a seasoned expert.

Ironically, the informaticist was told during his interviews that a CMIO they'd hired a few years ago had left, in part due to being overextended.  He was also told that some of the clinical IT problems he'd solved as a CMIO in the past Director of Informatics position were problems this organization had not been able to solve during the same time frame.

... Regarding underestimation, this organization appeared to have little idea of the difficulties they were getting into

... The informaticist had observed another indication that this organization ‘didn't know what they didn't know.’  He was informed that the organization had selected their EHR vendor prior to seeking a medical informatics expert. This implies they really did not understand what a medical informatics specialist does and can do, which is far more than being a tactical "EHR implementation assistant." ... The informaticist had been there, and had done that in his past role. He found it unrewarding then and actually had decided not to take the risk again, rejecting the new position, even before the organization decided they wanted an (effectively) part-time CMIO who also saw patients.

It was clear this organization, who the informaticist tried to “take to school” based on hard-earned expertise and extensive references on social issues in health informatics (e.g. on this website), felt they were the experts on what was best regarding CMIO background.   This may have been a dysfunctionality satirist Scott Adams once described like this:

Ignoring the Advice of Experts Without Good Reason

Example: Sure, the experts think you shouldn't ride a bicycle into the eye of a hurricane, but I have my own theory.

As I recall, an informatics physician much younger than I and with less experience was hired.  I do not know his fate.  However, I believe that my expertise at that time in EHR issues, and my no-nonsense stance on EHR safety and excellent management, might have averted or minimized the "inaccuracy" problems.

Therefore, it is also no surprise to me that the organization now relates the EHRs were causing significant charting inaccuracies.  It would also not surprise me if close calls and harm occurred. They apparently rode their bicycle into that hurricane, and doing so will usually produce a less than optimal outcome.

In summary, two recent articles on EHR-related lawsuits and operational problems, in my own personal experience, have potentially relevant "backstories" that may help provide context.  Those backstories relate primarily to what I've called "HIT amateurism" (e.g., faulty expertise evaluation) as well as HIT mismanagement (e.g., ignoring explicit internal warnings of dangers.)  I believe my experiences, and that of other such as Dr. Malin, are symptomatic of widespread health IT industry dysfunction.

I also believe that if this industry is ever to learn from its mistakes, it will only occur in the courtroom.  Being that the defense side is usually fierce, however, I am not sanguine needed learning will occur any time in the near future.

-- SS

Another Sign The EHR Experiment is Producing Undesired (But Predictable) Results: ED EHRs Produce No Efficiency Gains

Adverse results that run counter to the industry meme of healthcare IT exceptionalism are coming fast and furious.  It's hard for me to keep up with it all.

In addition to my post today "EHR Exceptionalism Debunked: Care Quality Variances Among VA Hospitals Suggest the True Value of Health IT Has Been Grossly Exaggerated" (http://hcrenewal.blogspot.com/2014/06/ehr-exceptionalism-debunked-care.html), here's another.

I won't comment on this very much:

http://www.jwatch.org/na34646/2014/05/30/electronic-charting-less-efficient-paper-charting

May 30, 2014 

Is Electronic Charting Less Efficient Than Paper Charting?
Daniel J. Pallin, MD, MPH reviewing Ward MJ et al. Ann Emerg Med 2014 Jun.

Operational performance was similar before and after computerization at 23 community emergency departments.

The federal government has provided $17 billion in incentives to computerize healthcare. The potential benefits include improved error checking, decision support, better billing, and more data for research. However, some research has suggested that going paperless adversely affects productivity.

To further examine this issue, investigators measured operational efficiency 6 months before and 6 months after implementation of an electronic health record system at 23 community emergency departments. No significant differences between the two time points were found in all efficiency factors that were measured, including time to provider, lengths of stay, walkouts, patient satisfaction, patients seen per provider per hour, and significant return visits.

Citation(s):

Ward MJ et al. The effect of electronic health record implementation on community emergency department operational measures of performance. Ann Emerg Med 2014 Jun; 63:723. (http://dx.doi.org/10.1016/j.annemergmed.2013.12.019)

One might seek to get better results for their $17 billion than, at best, parity to paper.

Finally, the reader comments at that post suggest ED physicians are the ones helping foot the bill.  As one commenter put it:

"Most quality EM docs won't let a record slow their care up front--especially when quaility metrics are staring us down on time until seen or discharge. So we just chart later and go home later."

We as patients certainly should not find a development like that desirable.

Note that in my work as a hospital Chief Medical Informatics Officer (CMIO) I recommended simple document imaging solutions for risk-laden ED's (where charts are usually short) to capture and make the paper chart content available anywhere/anytime ... not full-blown digital electronic medical records systems.

-- SS

Most quality EM docs won't let a record slow their care up front--especially when quaility metrics are staring us down on time until seen or dischage. So we just chart later and go home later. - See more at: http://www.jwatch.org/na34646/2014/05/30/electronic-charting-less-efficient-paper-charting#sthash.WCiV8FQg.dpuf

Most quality EM docs won't let a record slow their care up front--especially when quaility metrics are staring us down on time until seen or dischage. So we just chart later and go home later. - See more at: http://www.jwatch.org/na34646/2014/05/30/electronic-charting-less-efficient-paper-charting#sthash.WCiV8FQg.dpuf

Most quality EM docs won't let a record slow their care up front--especially when quaility metrics are staring us down on time until seen or dischage. So we just chart later and go home later. - See more at: http://www.jwatch.org/na34646/2014/05/30/electronic-charting-less-efficient-paper-charting#sthash.WCiV8FQg.dpuf

Most quality EM docs won't let a record slow their care up front--especially when quaility metrics are staring us down on time until seen or dischage. So we just chart later and go home later. - See more at: http://www.jwatch.org/na34646/2014/05/30/electronic-charting-less-efficient-paper-charting#sthash.WCiV8FQg.dpuf"

EHR Exceptionalism Debunked: Care Quality Variances Among VA Hospitals, Perhaps The Most Wired Hospitals in the World, Suggest the True Value of Information Technology in Healthcare Has Been Grossly Exaggerated

The Wall Street Journal somehow managed to get access to internal VA patient care quality data and published the following article:

http://online.wsj.com/articles/veterans-affairs-hospitals-vary-widely-in-patient-care-1401753437

Veterans Affairs Hospitals Vary Widely in Patient Care
Internal Records Show Facilities Such as Phoenix Have Far Higher Death Rates Than Peers
Thomas M. Burton and Damian Paletta
June 3, 2014

The Phoenix facility at the heart of the crisis at the Department of Veterans Affairs is among a number of VA hospitals that show significantly higher rates of mortality and dangerous infections than the agency's top-tier hospitals, internal records show.

The criticism that precipitated last week's resignation of VA Secretary Eric Shinseki has focused largely on excessive wait times for appointments across the VA's 150-hospital medical system.

But a detailed tabulation of outcomes at a dozen VA hospitals made available to The Wall Street Journal illustrates a deeper challenge: vastly disparate treatment results and what some VA doctors contend is the slippage of quality in recent years at some VA facilities.

Some of the discrepancies are stark, especially for an agency known for offering high-quality care in 50 states.

Yet, these are highly wired hospitals, perhaps the most highly wired in the world, with one EHR, VistA CPRS, allegedly among the best EHRs in the world developed over decades, and allegedly used by 100% of the clinicians.

How can this be? 

... Scott McRoberts, spokesman for the Phoenix VA Health Care System, said on Monday the database "is an internal measurement system to benchmark our improvement, and is not for public consumption."

The rate of potentially lethal bloodstream infections from central-intravenous lines was more than 11 times as high among patients at the Phoenix facility than it was at top VA hospitals, data from the year ended March 31, 2014, show.

... The findings come from a nonpublic VA database called Strategic Analytics for Improvement and Learning, known as SAIL. SAIL tracks procedure outcomes and ranks VA hospitals on a scale of five stars, the best, to one star, the lowest.

... In other examples of variations in care, the Atlanta VA Medical Center, a two-star hospital for quality, has more than three times the rate of central-IV infections than the average of five-star VA hospitals. Houston's VA hospital, ranked as a two-star hospital, had a 47% higher acute-care mortality rate than the five-star hospital rate.

A 47% higher acute-care mortality rate?

Hospitals are willing to go into tens of millions of dollars of debt and knowingly endanger patients (see e.g. "In Fixing Those 9,553 EHR "Issues", Southern Arizona’s Largest Health Network is $28.5 Million In The Red" at http://hcrenewal.blogspot.com/2014/06/in-fixing-those-9553-ehr-issues.html), and even attempt to stifle clinician conversation on drawbacks (e.g. see "Words that Work" at http://hcrenewal.blogspot.com/2013/10/words-that-work-singing-only-positive.html), to attain this technology on the stated reason of "improving patient safety" and "saving money."

Former ONC Chair Blumenthal stated explicitly in the NEJM article "The 'Meaningful Use' Regulation for Electronic Health Records", Blumenthal and Tavenner (10.1056/NEJMp1006114, July 13, 2010):

... EHRs will improve caregivers’ decisions and patients’ outcomes. Once patients experience the benefits of this technology, they will demand nothing less from their providers.

There is no doubt expressed there.

The industry itself has convinced the Federal Government to spend tens of billions of taxpayer dollars on this technology via HITECH incentives, and to start penalizing non-adopters via Medicare cuts.  The Government has bought into the hyper-enthusiasm hook, line and sinker, when the same money might have been spent elsewhere such as on care of the underserved...or on our Veterans.

Yet the data from a highly-computerized organization like the VA show that these systems cannot be all they are made out to be.  They do not automatically "improve caregivers' decisions and patients' outcomes." This is an on its face observation.

They are simply a tool, and in absence of appropriate human staffing, expertise, and conducive work environment, they mean significantly "less than advertised" in terms of improving care quality and outcomes. 

Finally, as the VA has gotten busier due to factors such as the return of veterans from recent wars, one must also ask if the loss of productivity, cognitive burden, and other factors caused by needing to interact with a computer rather than paper has made the computer systems more of an impediment to good care rather than a facilitating tool.

I think it's time the Government start to reform the industry hype surrounding this technology with far more sober predictions that would facilitate far more cautious implementations.

No "revolutions" are happening soon, at least not due to information technology in healthcare.

Finally, note that EHR oversell and exceptionalism has been going on for decades.   Donald A. B. Lindberg M.D. (http://www.nlm.nih.gov/od/roster/lindberg.html), Medical Informatics pioneer and Director of the U.S. National Library of Medicine at NIH, observed in 1969 that:

"The [second] reason for the dearth of significant success of computers in medicine has been the flagrant and consistent "over sell" of capability on the part of manufacturers and computer enthusiasts."  

Read his full quote by clicking on the page below, from Morris Collen's book "A History of Medical Informatics in the United States: 1950 to 1990."

Informatics pioneer Dr. Donald Lindberg's 1969 observations on health IT oversell.  Click to enlarge.

Sadly, nothing has changed in health IT, at least since the year we landed on the Moon.

-- SS

Words that Work: Singing Only Positive - And Often Unsubstantiated - EHR Praise As "Advised" At The University Of Arizona Health Network

When clinicians are told to promote a technology in no uncertain terms, that puts a chilling effect on critical thinking and discourse.  In effect, when under orders to only speak positively about a hospital or its technology, saying anything bad could very likely get clinicians labeled as 'troublemakers' or 'disruptive clinicians.'  Sometimes - in a sadly real example at Affinity Health - it may even get threats of having complaints plastered to one's forehead (see http://hcrenewal.blogspot.com/2013/07/hows-this-for-patient-rights-affinity.html), a threat answered to by a judge.

The 'disruptive' label usually does not have a good effect on one's evaluations and job (or, for doctors, even career) longevity.  See, for example, the resources at http://www.aapsonline.org/index.php/article/sham_peer_review_resources_physicians/ on sham peer review.

At University of Arizona Health Network (UAHN), clinicians are being told to promote the EPIC EHR.

The campaign is under the aegis of executives who know, should know, or should have made it their business to know the mayhem caused at other medical centers by EPIC and other major clinical IT systems (see for example query links http://hcrenewal.blogspot.com/search/label/EPIC and http://hcrenewal.blogspot.com/search/label/healthcare%20IT%20difficulties).

Here's what clinicians are bring told in the Oct. 3, 2013 "Weekly update for UAHN employees":

Words that Work 

Talking positively to our patients about our new Electronic Health Record system is important! Here are some key words and phrases you can use to emphasize the many benefits of the new system:

• Electronic health record (not ‘Epic’ or ‘EHR’)
• One comprehensive record
• Coordinated care
• Improves patient safety & quality
• Convenient, easy patient portal 
• Private and secure

Click here for more words and behaviors to inspire confidence in our patients (and ourselves) as we transition to this new system.

The link to "more words" produced this PDF:

"Words that Work" - If I worked there, I would be concerned that that using "words that don't work" about a project that probably cost hundreds of millions of dollars would likely injure my career.  Click to enlarge.

This is shameless.  Many of these claims are unsubstantiated or in significant doubt in the literature.

First:

They left out issues such as these:

• The software is tested and validated for safety by nobody, including traditional medical device safety testers.

• No postmarket surveillance for problems, either.

• Transparency about problems that can cause patient harm is severely impeded by systematic impediments to information flow (as per IOM's 2012 study of health IT safety at http://hcrenewal.blogspot.com/2012/03/doctors-and-ehrs-reframing-modernists-v.html, FDA via their leaked Internal Memo on HIT safety as at http://hcrenewal.blogspot.com/2010/08/smoking-gun-internal-fda-memorandum-of.html, the Joint Commission in their Sentinel Events Alert on Health IT as at http://hcrenewal.blogspot.com/2008/12/joint-commission-sentinel-events-alert.html, and others.)

• Problems known are only the "tip of the iceberg" (FDA, ECRI Institute), as at http://hcrenewal.blogspot.com/2010/02/fda-on-health-it-adverse-consequences.html and http://hcrenewal.blogspot.com/2013/02/peering-underneath-icebergs-water-level.html 

(Mar. 2014 addendum: add to that list the revelations of med mal insurer CRICO, that EHR technology has contributed to a substantial fraction of the medical malpractice suits in Massachusetts: http://hcrenewal.blogspot.com/2014/02/patient-safety-quality-healthcare.html)

Of the claims they do make:

• Efficient - see aforementioned links as well as "Common Examples of Healthcare IT Difficulties" at http://cci.drexel.edu/faculty/ssilverstein/cases/

• Convenient - as above.  According to whom?  Compared to what?  Pen and paper?

• Improves patient safety and quality - see IOM report post at http://hcrenewal.blogspot.com/2011/11/iom-report-on-health-it-safety-nix-fda.html .  We as a nation are only now studying safety of this technology, and the results are not looking entirely convincing, e.g. ECRI Deep Dive Study of health IT safety at http://hcrenewal.blogspot.com/2013/02/peering-underneath-icebergs-water-level.html.  171 health IT mishaps in 36 hospitals, voluntarily reported over 9 weeks, with 8 reported injuries and 3 reported possible deaths is not what I would call something that "improves patient safety and quality" without qualifications.

• The Cadillac of its kind - according to whom?

• Patients at hospitals using this system love it -  Do most patients even know what it, or any EHR, looks like?  Have they provided informed consent to its use?

• Exciting - clinician surveys such as by physicians at http://hcrenewal.blogspot.com/2010/01/honest-physician-survey-on-ehrs.html and by nurses at http://hcrenewal.blogspot.com/2013/07/candid-nurse-opinions-on-ehrs-at.html shed doubt on that assertion. 

(Addendum: also see what nurses think about these systems at "Another 'Survey' on EHRs - Affinity Medical Center (Ohio) Nurses Warn That Serious Patient Complications Only a Matter of Time in Open Letter", http://hcrenewal.blogspot.com/2013/11/another-survey-on-ehrs-affinity-medical.html)

• The best thing for our patients - again, according to whom?

• Sophisticated new system - "New"?  Not so much, just new for U. Arizona Health.  "Sophisticated", as if that's a virtue?  Too much "sophistication" is in part what causes clinician stress and burnout, raising risk; see this summary of a new, not-free JAMIA article "Electronic medical records and physician stress in primary care: results from the MEMO Study", J Am Med Inform Assoc amiajnl-2013-001875 at http://www.beckershospitalreview.com/healthcare-information-technology/the-relationship-between-emrs-and-physician-stress.html.   From that summary:

... Compared with physicians at clinics with low-function EMRs, physicians at clinics with moderate-function EMRs experienced significantly more stress and had a higher rate of burnout. Additionally, physicians at clinics with moderate- or high-function EMRs felt less satisfied with their current position overall.

and:

... Results also showed a significant relationship between time pressure and physician stress in the cohort with high-function EMRs, and only in this cohort, suggesting physicians at these clinics may be particularly pressured for time during patient encounters in the face of a large number of EMR functions. "This 'made sense' to us in thinking about the possibility that those in the high-use group had more to do in the EMR" [say the authors].

• Smartest program out there - "Smartest" meaning what, exactly?  According to whom?  Who performed the comparison?

• Streamlined - compared to what?

• Thank you for your patience - even if the effects on clinicians gets you or your loved ones maimed or killed?

• Safe and secure network - really?  No break ins, ever, considering multiple breach stories like those at http://hcrenewal.blogspot.com/search/label/medical%20record%20privacy?

• Keeping you informed is our priority - informed of what?

• Specially trained staff - like these:  http://hcrenewal.blogspot.com/2010/08/epics-outrageous-recommendations-on.html?

and this:

• Take Responsibility - I ask, should clinicians "take responsibility" for IT-related disruptions that impair care such as "use error" (as opposed to user error), i.e., what the National Institute of Standards and Technology has called operator error due to poor usability and other features of bad health IT?  (See "NIST on the EHR Mission Hostile User Experience" at http://hcrenewal.blogspot.com/2011/10/nist-on-ehr-mission-hostile-user.html.)  What about "glitches" and bugs that corrupt or lose data?  Should clinicians also 'take responsibility' for those?  (See for example the posts on the wild things that happen when IT malpractice leads to clinical mayhem at http://hcrenewal.blogspot.com/search/label/glitch.)

It appears to me that this vendor is using its client hospitals' management to enforce an "acceptable point of view" clinicians must proffer to patients about EHRs (which they must call "health" records), despite well-known contradictory findings.  This is, in effect, forced marketing of a device.

Trying that for a drug or a conventional medical device (e.g., a particular stent) would be on its face unethical and likely illegal.

Finally, critical thinking is what keeps patients alive and safe.  Marketing measures like this (some might call it "propaganda"), espousing and enforcing 'EHR exceptionalism', in my opinion, damage critical thinking and expression, and are thus unacceptable to push on clinicians and on patients.

I add that requiring clinicians to promote deceptive propaganda the clinicians themselves know is untrue, from painful experience, is degrading, intimidating and destroys morale.  It is axiomatic that clinicians (or anyone) operating under such conditions cannot perform at their best.

Thus the management geniuses who came up with these instructions (if not outright vendor-ghostwritten as at the Aug. 2012 "Health IT Vendor EPIC Caught Red-Handed: Ghostwriting And Using Customers as Stealth Lobbyists", http://www.tinyurl.com/epic-stealth) are by their actions increasing risk of patient harm.

The nurses' unions at at http://hcrenewal.blogspot.com/2013/07/rns-say-sutters-new-electronic-system.html have it right, in my view:  complain about the disruptions this technology causes, and complain loudly, if at the very least to make sure the problems are out in the open.

-- SS

Note: also see the followup Nov. 2013 post "We’ve resolved 6,036 issues and have 3,517 open issues: extolling EPIC EHR virtues at University of Arizona Health System" at http://hcrenewal.blogspot.com/2013/11/weve-resolved-6036-issues-and-have-3517.html

Once Again, on IBM Watson, Cybernetic Miracles and Reductionist Views of Medicine

The cybernetic miracle shamans are at it again:

IBM's Watson set to tackle health insurance, takes 'Diagnosis for $1,000'
via Engadget by Billy Steele on 9/12/11

After tackling your tech support woes, the famed Watson is moving on to mop up the health insurance industry. That's right, the IBM showstopper we all know and love for trouncing trivia kings on Jeopardy has been hired by one of the largest health insurance company's in the US. WellPoint Inc. [yes - this WellPoint, a company that HC Renewal has referenced not just a few times - ed.] will make use of the system's breakneck speed and healthcare database alongside patient records -- allowing the supercomputer to guide treatment options and prescribe medicines. [Med Mal plaintiff's attorneys: take note - ed.] Once implemented, data will be combined from three sources in a matter of seconds: a patient's chart / records from a doctor, the insurance company's patient history and the medical knowledge that Watson already possesses. A pilot program will roll out next year to a number of cancer facilities, academic medical centers and oncology practices. No word yet on when The Watson School of Medicine will start accepting applications.

I wrote on these issues quite seriously back in January 2011 at this post: "IBM's Watson, Jeopardy, and Revolutionizing Medicine."

I wrote on a somewhat sarcastic note at this March 2011 post: "Here Comes the Judge! A Quick Thought on Cybernetic Medicine: Why Can't Computers Also Do Law?"

My views are unchanged since then.

Roy Poses adds a good point:

• Where are the clinical trials? There is nothing here about actually doing clinical trials to test if Watson is accurate enough to do more good than harm before rolling this out. [As Scot has said a zillion times] This is an experimental technology akin to a new medical device that should be tested as rigorously as any new device or drug should be tested before it is used on patients.

To which I further add:

• Where is the patient informed consent?
  

-- SS

Addendum:

Perhaps Watson should better replace our "they're all the best and brightest" hospital CEO's and managers.

IBM's Watson, Jeopardy, and "Revolutionizing Medicine"

In the news recently was a story about a new supercomputer doing amazing things. In a technological tour de force, IBM's Watson supercomputer research project has reached a milestone, beating a group of contestants in the TV gameshow Jeopardy.

The TV game show "Jeopardy"

Will predictions of advancements in medicine follow? Predictably so:

IBM's Watson Supercomputer Beats Humans in Jeopardy Practice Match
eWeek.com
By: Fahmida Y. Rashid
2011-01-13

Watson, IBM's latest DeepQA supercomputer, defeated its two human challengers during a demonstration round of Jeopardy on Jan. 13. The supercomputer will face former Jeopardy champions Ken Jennings and Brad Rutter in a two-game, men-versus-machine tournament to be aired in February.

However, the Jeopardy match-up was not the "culmination" of four years of work by IBM Research scientists that worked on the Watson project, but rather, "just the beginning of a journey," Katharine Frase, vice president of industry solutions and emerging business at IBM Research, told eWEEK.

Supercomputers that can understand natural human language—complete with puns, plays on words and slang—to answer complex questions will have applications in areas such as health care, tech support and business analytics, David Ferrucci, the lead researcher and principal investigator on the Watson project, said at the media event showcasing Watson at IBM's Yorktown Heights Research Lab.

Watson analyzes "real language," or spoken language, as opposed to simple or keyword-based questions, to understand the question, and then looks at the millions of pieces of information it has stored to find a specific answer, said Ferrucci.

This is undoubtedly a remarkable accomplishment.

Indeed, accompanying the announcements we are also seeing predictions that such supercomputers "will have applications in health care."

Indexing of the medical literature, and data mining (for better or worse) from free text come to mind.

However, the current irrational exuberance about healthcare IT in 2011 is based on several misconceptions. This leads to predictions such as this ...

The technology has to process natural language to understand "what did they mean" versus "what did they say," which has a lot of implications in the health care sector, said Frase. Patients are not using the terms doctors learned in medical school to describe their ailments, but more likely the terms they picked up from their parents growing up, she said.

"Patients are not using the terms doctors learned in medical school to describe their ailments"?

What medical school(s), exactly, are being spoken of here?

It seems as if IT folks think medicine was invented just yesterday. In fact, in medical school, internship, residency and practice we learn all about that, and learn how to 'translate' that information or use it to elicit more information as needed in order to provide care. I'm not sure a multimillion dollar supercomputer is needed for that ...

... and related "platform database" predictions such as this:

... A Watson-like system can take that information and co-relate it against all the medical journals and relevant [who decides that? - ed.] information, and say, "Here's what I think [think? -ed] and why," while showing its evidence for how it came up with the conclusion, according to Frase.

(Actually, computers don't think. A more correct statement would be "here are the results of the algorithms that your faithful machine has crunched, using the medical literature as input.")

That's quite naïve and idealistic with regard to actual medical decision making. It is a computer technician's oversimplified, reductionist, amateur view regarding biomedicine, a domain of often wicked complexity.

As I intimated above, one key issue is what is "relevant" with regard to information.

Consider the issue of the medical literature suffering from numerous conflict of interest and dishonesty-related phenomema making it increasingly untrustworthy, as pointed out by Roy Poses in a Dec. 2010 post "The Lancet Emphasizes the Threats to the Academic Medical Mission", at my Aug. 2009 post "Has Ghostwriting Infected The "Experts" With Tainted Knowledge, Creating Vectors for Further Spread and Mutation of the Scientific Knowledge Base?" and elsewhere on this blog.

Then too, there are plausibility issues in medical research, as expressed in the paper "Why Most Published Research Findings Are False", John P. A. Ioannidis, PLoS Medicine 2(8): e124, 2005. Dr. Ioannidis observes:

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.

Dealing with these very real-world issues in patient care requires nothing less than human judgment borne of experience, critical thinking skills (emphases on "thinking", which computers cannot do, sorry all you HAL-9000 and M5 fans) and intuition to manage. "Garbage in, garbage out" applies in the extreme.

(The Ultimate Computer, Dr. Richard Daystrom's mid-23rd century Multitronic unit, M5. Fun to watch it beat up on starships Excalibur, Hood, Lexington and Potemkin due to malfunction. Even more fun to watch William Shatner talk it into cybernetic suicide via guilt, and yes, as a teen I knew all the ST trivia cold, but I think sci-fi has gone too far into people's heads.)

There are also other computer-vs-human mind issues at play.

First, let's look at Watson. Just to play the TV quiz show game show Jeopardy, a game largely about the knowledge and recall of trivia, it took the following:

• Watson is a breakthrough human achievement in the scientific field of Question and Answering, also known as "QA." The Watson software is powered by an IBM POWER7 server optimized to handle the massive number of tasks that Watson must perform at rapid speeds to analyze complex language and deliver correct responses to Jeopardy! clues. The system incorporates a number of proprietary technologies for the specialized demands of processing an enormous number of concurrent tasks and data while analyzing information in real time.

• An entire team of "algorithm specialists" writing algorithms specialized to the purpose of "the special question processing" needed to answer Jeopardy questions, such as: "On Sept. 1, 1715 Louis XIV died in this city, site of a fabulous palace he built", and "To marry Elizabeth, Prince Philip had to renounce claims to this southern European country's crown."

• Watson took about four years to build and is made up of 10 racks of [multiprocessor - ed.] IBM servers running Linux with 15 terabytes of memory [presumably fast-access RAM, not hard disk -ed.], the report said. [as per the LA Times - ed.] Over those four years, IBM has filled Watson with about 200 million pages' worth of information from encyclopedias, dictionaries, books, news and movie scripts, the AP said.

A rack-mountable server ("blade").

Racks of servers, each with multiple advanced CPU's

Racks of servers in IBM's Watson supercomputer, all prettied up, HAL 9000-style

Second, it will be many years indeed before even the current Watson QA capabilities can be tailored to a domain as complex as biomedicine and made widespread for the hundreds of thousands of practicing physicians and the much larger number of allied healthcare professionals in the U.S. or worldwide.

Third, there's this from the same LA Times article linked above:

• Like its human competitors, Watson won't have Internet access [or access to anything outside its immense local storage - ed.] during the games, so Googling an answer won't be an option, the report said.

I see a significant degree of machine-human unfairness right there. A computer has 100% reliable access to information in its storage media. The human mind does not. That's appropriate for a TV game show that tries to test a persons' knowledge of trivia, but that's not how medicine works.

What if the match were made more fair, giving the humans error-free access to the same information Watson has stored in its own 15 terabytes and 200 million pages?

Fourth, medicine is not about recalling trivial factoids of information based on parsing natural language queries in the "puzzle format" of Jeopardy. As I wrote here and here, medicine is not a platform database information retrieval problem. (I would argue medicine, except in simple cases, is in large part a matter of filtering the irrelevant, unlikely, and unreliable, of which there is an exponentially increasing amount, from information relevant to the subtleties of a complex medical situation.)

Certainly, today's clinical IT will make little dent in healthcare quality and error reduction, as those issues are not in majority due to record keeping problems of paper vs. electronic, as I pointed out in my Dec, 2010 post "Is Healthcare IT a Solution to the Wrong Problem?". The expectations for today's health IT are grossly exaggerated.

How about expert systems technology such as Watson? Regarding NLP and fact retrieval 'tours de force' like IBM Watson, medicine is about cognition, about human judgment born of experience in dealing with ambiguity not just of language but of observations, findings, lab data, image interpretation, etc., about human intuition and assemblage and integration of a huge amount of disparate information in ways not well understood even by its practitioners. The end result is not just recall of a piece of information.

I consider a statement such as:

... a Watson-like system can take that information and co-relate it against all the medical journals and relevant information, and say, "Here's what I think [actually, as mentioned previously, here's the results of the algorithms - ed.] and why"

... to imply just as grandiose a valuation to the technology as the statements I heard a decade ago about the health IT of the day - or even today - "revolutionizing medicine."

I'm not even sure such a capability would be very useful; we already have DXplain developed by domain experts over decades, and that's not had a major impact on healthcare to date.

The real breakthrough will be when a cybernetic expert system can take, say, cases from the Case Records of the Massachusetts General Hospital in the NEJM verbatim, and compete as a peer (e.g., as a peer not recognized to be a machine) with a round-table panel of expert physicians with facile access to the medical literature (e.g., PubMed) on the differential diagnosis, how to establish the diagnosis and to rule out others, the treatment strategies, and the likely outcomes, and then participate in the care.

When this happens (call this the "NEJM Turing Test"), and when such capabilities are affordable and widely available, then medicine will have been revolutionized.

Of course, the enormity of the hardware and the algorithmic advances required to make a truly "revolutionary" tool such as this are obviously staggering. Considering that it takes 10 racks of multiprocessor IBM servers with 15 terabytes of memory and a team of varied domain experts writing algorithms for several years to accomplish the NLP advances and lookups to answer Jeopardy-style trivia questions, one can only imagine what a truly useful cybernetic medical assistance system would look like.

It should also be remembered that Watson does not think. Humans do. Cybernetic Jeopardy and chess playing accomplishments notwithstanding, I believe a machine even close to passing a "NEJM Turing test" will be a long time in coming. Until then, we should be encouraging better support for human physicians struggling to use their medical expertise in a sea of bureaucracy, stress and overwork (part of which will increasingly be a struggle with mission hostile health IT).

Finally, far off as I believe it to be, I do think the hundreds of billions of dollars being devoted to today's health IT would be better devoted to developing a "Dr. Watson" that can pass a medical Turing test as described above, than deploying mission hostile, primitive HIT and developing an unsustainable mega-bureaucracy to support it as in my post here.

This is not to minimize the wonderful accomplishments of the Watson team. I just wish predictions of cybernetic miracles in medicine would be held in abeyance after the lessons of the past fifty years of computers in medicine.

In the meantime, perhaps the current Watson might be useful in remediating our very sclerotic and moribund "mainsteam media," especially in the domain of politics. Its reporters and writers can certainly use cybernetic help in their fact-checking and logic, Jeopardy-style, far more so than physicians.

-- SS

Feb. 23, 2011 addendum:

Gevalt!

From the article "IBM's Watson could usher in new era of medicine", Sharon Gaudin, Computerworld, February 17, 2011:

Jennifer Chu-Carroll, an IBM researcher on the Watson project, said the computer system is a perfect fit for the healthcare field ... Think of some version of Watson being a physician's assistant," Chu-Carroll said. "In its spare time, Watson can read all the latest medical journals and get updated. Then it can go with the doctor into exam rooms and listen in as patients tell doctors about their symptoms. It can start coming up with hypotheses about what ails the patient.

Gevalt indeed ... incredible irrational exuberance. We can easily go from Jeopardy to Medicine ... and then to the Moon, in a hot air balloon! (The moon is up, hot air balloons go up, what's the problem?)

How many patients has Chu-Carroll seen lately?

(Her bio at http://researcher.ibm.com/researcher/view.php?person=us-jencc was initially down at this moment; "The server at researcher.ibm.com is taking too long to respond." Watson must be taking a nap. Archive.org says "The connection has timed out. The server at web.archive.org is taking too long to respond." Just like a doctor!)

Anyway, it now appears:

I am a Research Staff Member at IBM T. J. Watson Research Center. I also manage the Knowledge Structures group which focuses on improving advanced search technology through the use of natural language processing and machine learning techniques. Prior to joining IBM in 2001, I spent 5 years as a Member of Technical Staff at Lucent Technologies Bell Laboratories. My research interests include question answering, semantic search, natural language discourse processing, and spoken dialogue management.

Please, please, please, computer scientists: STOP MAKING THESE PREDICTIONS OF CYBERNETIC MIRACLES JUST AROUND THE CORNER. YOU'VE BEEN DOING IT SINCE THE VACUUM TUBE-BASED MACHINES.

STOP! PLEASE!

(Maybe first they could give us a computer that does a perfect, lucid, coherent, fluent translation of, say, Russian-to-English, another promise made since the 1950's "Robby the Robot" years?)

Robby the Robot in "Forbidden Planet." Lost in Space fans will remember Robby as doing battle with the Class M-3 Model B9, General Utility Non-Theorizing Environmental Control Robot ("Danger, Danger, Will Robinson") as well!

I think the essay in today's WSJ by UC Berkeley philosopher John Searle is also apropos: Watson Doesn't Know It Won on 'Jeopardy!'

-- SS

Addendum: See my Wall Street Journal Letter to the Editor on these matters at this link.

I wrote on a somewhat sarcastic note at this March 2011 post: "Here Comes the Judge! A Quick Thought on Cybernetic Medicine: Why Can't Computers Also Do Law?"

Also see my Sept. 2011 followup post "Once Again, on IBM Watson, Cybernetic Miracles and Reductionist Views of Medicine."

-- SS

UK: ISO draft standards for the development, manufacture and deployment of healthcare IT focus on SAFETY

The UK's NHS has not had the best of success to date implementing national health IT, as indicated by reports here and here, for example.

However, they have appeared to have learned from their mistakes and in fact are on the way to being far ahead of the U.S. in terms of understanding what it truly takes for HIT to be efficacious - and perhaps even more importantly, as safe as possible.

From an informatics colleague who informed me of these developments:

The UK has recently adopted the ISO draft standards for the development and deployment of HIT. They don't go as far as premarket approval, but do require vendors to develop and deliver to healthcare organizations a formal hazard assessment for their products, require both to continually update their risk assessments, and require care delivery organizations to have an explicit process for identifying & mitigating risks, and formally accepting (or not) the residual risks that remain. The thinking is these standards will be adopted across the EU once the ISO approval process is completed.

These two remarkable documents are available from the UK's NHS:

http://www.isb.nhs.uk/documents/isb-0160/dscn-18-2009

"Health informatics — Guidance on the management of clinical risk relating to the deployment and use of health software"
Formerly ISO/TR 29322:2008(E)
DSCN18/2009

and

http://www.isb.nhs.uk/documents/isb-0129/dscn-14-2009

"Health Informatics — Application of clinical risk management to the manufacture of health software"
Formerly ISO/TS 29321:2008(E)
DSCN14/2009

From the first of these, the overall intro:

ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies (ISO member bodies). The work of preparing International Standards is normally carried out through ISO technical committees. Each member body interested in a subject for which a technical committee has been established has the right to be represented on that committee. International organizations, governmental and non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization.

Then on to matters at hand:

Introduction

The threat to patient safety

There is mounting concern around the world about the substantial number of avoidable clinical incidents which have an adverse effect on patients, of which a significant proportion result in avoidable death or serious disability, see references [1], [2], [3], [4], [5] and [6]. A number of such avoidable incidents involved poor or "wrong" diagnoses or other decisions. A contributing factor is often missing or incomplete information, or simply ignorance, e.g. of clinical options in difficult circumstances or of the cross-reaction of treatments (a substantial percentage of clinical incidents are related to missing or incomplete information).

It is increasingly claimed that information systems such as decision support, protocols, guidelines and pathways could markedly reduce such adverse effects.

[As I have written in many places such as here and here, this may or may not be true regarding today's commercial healthcare IT as it is currently designed and deployed. Evidence supporting the assertion, especially robust studies such as randomized controlled clinical trials, is scarce, and evidence contradicting it is growing. The technology remains experimental - ed.]

If for no other reason – and there are others – this is leading to increasing deployment and use of increasingly complex health software systems, such as for decision support and disease management. It can also be anticipated that, due to pressures on time and to medico-legal aspects, clinicians will increasingly rely on such systems, with less questioning of their "output", as a "foreground" part of care delivery rather than as a "background" adjunct to it. Indeed, as such systems become integrated with medical care, any failure by clinicians to use standard support facilities may be criticised on legal grounds.

Increased use of such systems is not only in clinical treatment but also in areas just as important to patient safety, such as referral decision-making. Failure to make a "correct" referral, or to make one "in time", can have serious consequences.

Economic pressures are also leading to more decision support systems. The area of generic and/or economic prescribing is the most obvious, but achieving economy in the number and costs of clinical investigative tests is another.

Thus the use of health software and medical devices in increasingly integrated systems, e.g. networks, can bring substantial benefit to patients. However unless they are proven to be safe and fit for purpose they may also present potential for harm or at least deter clinical and other health delivery staff from making use of them, to the ultimate detriment of patients. Annex A provides some examples of the potential for harm.

Harm can of course result from unquestioning and/or non-professional use, although the manufacturers of health software products, and those in health organizations deploying and using such products within systems, can mitigate such circumstances through, for example, instructions for use, training and on-screen presentation techniques, guidance, warnings or instructions.

Some of these system deficiencies are insidious, may be invisible to the end user [an obviously perilous situation - ed.] and are typically out of the sole control of either the manufacturer or the deploying health organization.

The reports note the obvious, something that the health IT vendors' contractual gag clauses and secrecy in the health IT industry make difficult to rigorously evaluate:

A necessary pre-cursor for determining and implementing controls to minimize risks to patients, from a health software systems that is manufactured and then deployed and used within a health organization, is a clear understanding of the risks which the deployed system might present to patients if malfunction or an unintended event were to occur, and the likelihood of such a malfunction or event causing harm to the patient.

These risks cannot be properly evaluated in an industry where the flows of information are dominated by the vendors.

Some examples of potential for harm, from annex (appendix) A, will likely sound quite familiar to readers of Healthcare Renewal:

• Patient (mis)identification
• Inadvertent accidental prescribing of dangerous drugs (such as methotrexate)
• Incorrect patient details retrieved from radiology information system
• CT and MRI images could not be seen after being moved to PACS
• Drug mapping error
• Pre-natal screening risk computation errors
• Radiotherapy errors
• Slack security

I especially note the following in the first document (on deployment):

5.3 Competencies of personnel

Persons performing risk management tasks will need to have the knowledge, experience and competencies appropriate to the tasks assigned to them. This will need to include, where appropriate, knowledge and experience of the particular health software systems (or similar health software products) and applications, the technologies involved and risk management techniques. This should include appropriate registered clinical input throughout the process. Appropriate competency and experience records will need to be maintained.

Clinical risk management tasks can, and should, be performed by a project team that contains representatives of each of the functions that are involved in deploying and subsequently using the health software systems or system, with each contributing their specialist knowledge to build both awareness and consensus. Of particular importance will be clinical input from clinicians who are familiar with the practical realities of the environments within which the software system will be used and the clinical processes to which the software system is directed.

Emphasis on the last sentence is mine. At a time when U.S. CIOs and health IT "talking heads" still find the need to write touchy-feely "Master of the Obvious" articles extolling the virtues of permitting clinicians 'input' into health IT projects, usually under the aegis of unempowered "Directors of Informatics" or "Chief Medical Information Officers" (a.k.a. Directors of Nothing and Chiefs of Nothing, with no true executive presence or authority), the latter direct, definitive sentence is refreshing.

Miracle of miracles, even postmarketing surveillance is covered (the pharma and medical device industries have been mandated by regulators to conduct such studies on their products for decades):

11 Post-deployment monitoring

Both manufacturers and organizations deploying and using health software and other products within systems, have a business need to establish, document and maintain a process to collect and review information about the clinical safety performance of the products and system in the post-deployment phase, at least to help manage their liabilities but also to enable them to optimize their products and systems.

There is much more in these documents.

Download and read the PDF's. I will have more to say in future posts, but thank god someone is considering the risks to patients of this technology, touted as universally beneficent by health IT exceptionalists, in a serious manner.

Now if only we can import this thinking into the United States.

-- SS