On an EMR Forensic Evaluation by Professor Jon Patrick from Down Under: More Thoughts.

Note: the reports to which this post refers are available as a set of PDF's from the University of Sydney, NSW, Australia at this link (12 Mb, .zip folder), or at "A study of an Enterprise Health information System" at http://sydney.edu.au/engineering/it/~hitru/index.php?option=com_content&task=view&id=91&Itemid=146.

March 6, 2011 addendum. Also see my new post "What to do about the state of the ED EHR in NSW?"

Over ten years ago now, 1999 in fact, I started my healthcare IT difficulties website.

That site, "Contemporary Issues in Medical Informatics: Common Examples of Healthcare Information Technology Difficulties" now resides on a Drexel University server in Philadelphia, Pennsylvania USA at this link, and used as a teaching resource.

I started the site after observing, for lack of a better description, "crazy stuff" in the commercial healthcare information technology sector.

Crazy stuff such as EMR systems for ICU's that crashed regularly and spread pathogens around, EMR's for invasive cardiology cath labs that were an informational jumble and abyss and that also issued regular General Protection Faults and died, lack of Medical Informatics expertise (and actual disdain for it) in healthcare IT projects, grossly incompetent IT leaders, and hospitals uncritically and enthusiastically buying these products as if they were a plug and play, proven technology.

To make matters worse, I also observed executives expressing a hostile indifference to glaring deficiencies.

My observations about health IT and about responses to my counsel brought to mind the biblical passage:

"Give not that which is holy unto the dogs, neither cast ye your pearls before swine, lest they trample them under their feet, and turn again and rend you." - Matthew 7:6

I was never afforded the opportunity to perform a forensic analysis of the internals of these systems, being that I was not allowed to obtain the software or "schematics" of the data structures. In fact, to make the cardiology system usable, I ordered the IT staff to simply discard the internal data dictionary, relational data structures, input screens, and analytic routines, and rebuild them - from scratch - using a proper Medical Informatics-based, cardiology domain expert-driven, iterative and incremental (agile) approach.

The result was a resounding success, and was described in a written report as "exceptional" by national specialty association reviewers invited to evaluate the effort.

(A success for which, I might add, I and my enlightened executive sponsor were paradoxically demonized by the IT department and hospital executives; cf. Matthew 7:6.)

Now, an Australian researcher of considerable computer and database expertise, Professor Jon Patrick at the University of Sydney, has put considerable ink to a forensic evaluation of the internals and external reactions to an EMR system "built in America", Cerner Firstnet.

Professor Patrick holds a Ph.D. from Monash University. He came to the University of Sydney from Massey University, New Zealand, where he held the foundation Chair of Information Systems. Professor Patrick won Australia's national Eureka science prize in 2005 for developing a natural language processing system that detected financial scams in web pages at the behest of the Australian Government.

FirstNet is an ED EHR that government officials decided must be installed in ED's of public hospitals throughout the Australian state of New South Wales. Promo material below, click to enlarge:

FirstNet promo material, page 1. Click to enlarge. (Hmm ... Is there a subliminal message in the picture of the doctor and nurse?)

FirstNet Promo material, page 2. Click to enlarge.

The initiative's been underway for several years, and the result is a group of apparently very unhappy Wal-Mart shoppers. (I guess the correct line would be "unhappy Big W shoppers" for those Down Under.)

Prof. Patrick had written a preliminary essay on the issue entitled, in rhetorical question-style, "The Story of the Deployment of an ED Clinical Information System: Systemic Failure or Bad Luck?" back in 2009. He apparently found himself in considerable hot water for doing so due to 'pushback' as I described at "Academic Freedom and ED EHR's Down Under: An Update". However, his university stood by him in defense of academic freedom (and of the sanctity of those in the healing arts, I might add).

He's spent the intervening time expanding the analysis of the ED clinical system and its deployment considerably, right down to the fine nuances of relational database design in complex domains (such as biomedicine).

As I wrote in an initial post "A Study of an Enterprise Health information System - Finally, an Informatics Scientist Does A Rigorous Review of a Commercial EHR System, by Cerner", the TOC of his new analysis are these (the files are available as a .zip archive at this link):

3.0 Part 0 - Executive Summary
3.1 Part 1 - A Critical Essay on the Deployment of an ED Clinical Information System ‐ Systemic Failure or Bad Luck? First published here in Oct 2009, revised Dec 2009.
3.2 Part 2 - Discussions with ED Directors: Are we on the right track?
3.3 Part 3 - Discussions with Software Performance Experts.
3.4 Part 4 - Conceptual Data Modelling.
3.5 Part 5 - Database Relational Schema and Data Tables.
3.6 Part 6 - Coalescing the Analyses of the ER Diagrams, Relational Schemata and Data Tables.
3.7 Part 7 - The Integrated Assessment.
3.8 Part 8 - Future HIT Regulation Proposals.
3.9 Part 9 - Ockham's Razor of Design. Published at the IHI conference, Nov 2010 Washington.

I have been reading these sections, and have found the technical sections (parts 4-6) highly informative about a major suspicion I've held for many years.

I suspected chaos in the health care IT software engineering process, with inadequate attention to quality, rigor, fine detail, resilience engineering, talent management and other practices essential in development of mission critical products of any type.

Prof. Patrick's forensic analysis, while not proof of my concerns, certainly supports them. If Boeing produced aircraft with malfunctioning engines, broken seats, defective flaps, tires that blew on landing, and rust right out of the factory (like the Chevy Vega of old?), one might suspect the development and manufacturing environment could be substantially problematic.

The theme of apparent violations of fundamental precepts of relational database design run consistently through his analysis of the FirstNet product.

Without getting too technical (which I can, having written successful relational database-based clinical information systems of considerable complexity for challenging environments, with novel user interaction design besides), I see evidence of developmental chaos.

Examples: primary key-foreign key inconsistencies and problematic usages ("keys" are flags used to link sets of information about some object or entity, such as a patient to their diagnoses or meds), internal field nomenclature faux pas (there are best practices on how to do this to enhance software quality and maintenance), cryptic documentation , "stale bits" (old code and data) from past iterations remaining to create "glitches", unreliabilities and new problems, and other technical sins apparently abound.

These can be read about in sections 4-6 of Prof. Patrick's analysis. The issues can be summarized as he did in the part 6 Abstract:

Consistent weaknesses in sections of the Millenium clinical information System (CIS) are revealed in the combined study of the ERD (entity-relationship diagram), logical schema and the data tables. PK (primary key, i.e., unique identifier) values are not always defined unambiguously at the design level and data tables reveal inconsistencies in declarations and data validation. There is evidence that keys are managed by software within the application rather than by the in-built functions available in the database management system leading to less confidence in data integrity.

He goes on to relate:

The [technical design] weaknesses in terms of clinical work practices, that have been identified are only likely to show up in occasional circumstances with a combination of processing and data values separated in time. [In other words, the resulting errors are unpredictable, and depend on variable factors about the patient's data and user's attempted actions that cannot be predicted ahead of time - ed.] Staff are not likely to associate one instance of missing or mis-processed data with another. This spasmodic nature tends to lull staff into a false sense of security that the mis-processing is either inconsequential or an accident of their own making. We recommend that each and every mis-processing experience be recorded as accurately as possible so that appropriate computational forensic analysis can correctly identify if weaknesses in the underlying technology have been the source.

These are dead-serious matters, literally. One's well being in an ED should not depend on random chance. If you are the "lucky patient" who Wins the Lottery or Hits the Jackpot on health IT mis-processing, or whose clinicians are distracted by user experience flaws, "workarounds", demoralization or other issues, you might end up maimed - or in the grave.

The ED EHR Slot Machine. Click to enlarge. You've hit the ED EHR mis-processing jackpot! Perhaps today is a good day to die...

I do not believe mission critical software for, say, avionics, or for implantable medical devices, suffers such sloppiness. (In part due to regulation, which health IT lacks entirely in the U.S.).

The UK, having their own HIT issues (see my Aug. 2010 "Battle of Britain" post at this link), apparently learned something, as evidenced in:

Health informatics — Guidance on the management of clinical risk relating to the deployment and use of health software. UK National Health Service, DSCN18 (2009), formerly ISO/TR 29322:2008(E).

and

Health Informatics — Application of clinical risk management to the manufacture of health software. UK National Health Service, DSCN14 (2009), formerly ISO/TS 29321:2008(E).

That said, I will now comment in more detail on a part of the analysis readily understandable by 'database laypeople': part 2, "Discussions with ED Directors: Are we on the right track?" (again, probably a rhetorical question).

In this section, candid discussions were held with the Directors of seven Emergency Departments in New South Wales public hospitals assessing the impact of the introduction of the FirstNet information system into their ED's. The effort has been ongoing for approximately the last 5 years.

Numerous themes remind me of my own observations as in my aforementioned Drexel health IT difficulties site:

The implementation processes of the HSS [governmental Health IT support, a.k.a. Health Support Services - ed.] were criticised for refusing to acknowledge the validity of complaints, failing to fulfil promises, creating an ineffective change process, refusing to consult clinicians, using strategies to disenfranchise participation by clinical staff, and introducing a technology that doesnʼt fit their needs.

All of these themes are familiar to me, and are representative of the phenomena of non-clinician, IT-centric arrogant ignorance, paternalism, leadership-pyramid inversion (i.e., the facilitators thinking and acting as if they are the enablers of healthcare), playing nasty politics with clinicians to avoid work, and minimizing job and results-evaluation discomfort for those lucky enough to secure cushy IT jobs in health IT support.

In determining the clinical documentation needs of staff, the Directors claim that the HSS ignores the needs of staff. Directors report over-supply of irrelevant information and under-supply of needed information in the clinical interfaces. ["Legible gibberish" - ed.] The environment consists of counter-intuitive interfaces where data is entered by one person in one part of the system so that it is not discoverable by another person.

The interfaces have inappropriate sizing of objects, confusing functions, redundant steps, unused functions and cluttered interfaces. These difficulties have resulted in increased time usage on the system resulting in decreased time with patients for no gain in administrative or clinical outcomes. Staff minimise their use of the system to as little as possible with work arounds being constantly developed and improved. Staff morale has been clearly degraded with accompanying loss of respect for the HSS and more generally NSWHealth’s authority.

These concepts are described and illustrated in my multi-part essay on the healthcare IT mission hostile user experience at this link. They represent major deviations from good information science, information presentation and human-computer interaction (HCI) precepts.

... Workarounds in using the system are the most obvious tangible response of staff to the functions of the system they consider unsatisfactory. The key aspect of workarounds is that they constitute a subversion of the policy processes created by the software that the staff are not prepared to collaborate with. Some of these strategies may even compromise the legal status of the records in the system: such as not signing documents, unrecorded alterations to documents, and test results not attached to patient records.

... Another form of staff protest workaround is the strategy by staff to avoid using the system by either having other people [presumably underlings - ed.] do the work on the system, inserting minimal amounts of information thereby reducing the value of the information and passing information to other staff verbally.

Workarounds to IT obstacle courses and booby traps, as noted by Koppel, Wetterneck, Telles & Karsh in "Workarounds to Barcode Medication Administration Systems: Their Occurrences, Causes and Threats to Patient Safety", J Am Med Inform Assoc. 2008 Jul-Aug;15(4):408-23, increase, not reduce, the risk of EHR-mediated medical errors. I wrote about their findings at "Business v. clinical computing: Workarounds to Barcode Medication Administration Systems" at this link.

It should be kept in mind that these are mission-critical systems for use in a fast-paced ED, not tracking systems for widgets - or for lab rats.

The lack of appropriate reporting functionality of the system has had a serious impact on the critical work of the Directors on process improvement ... It was evident in talking to the Directors that they have antennae highly tuned to the processes happening in their departments and the public health issues that emerge from their patients.

On a daily, weekly and monthly basis they review single cases, collections of common cases, and variations in established disease profiles to understand the success of their work and to detect emerging new trends or potential new disease outbreaks. At an administrative level they are asked to review cases either because of the return of new test results or due to complaints or reviews from other bodies.

The FirstNet installation has removed all the reporting functionality the directors had in their previous system EDIS while destroying their information sources for process improvement, and their mechanisms for creating and collaborating in research projects. This, in turn, has led to a loss of motivation to enter data further degrading the value of the data held within the system.

Reducing the value of a tool to people, and decreasing their ability to perform their jobs (especially when they take great professional responsibility and pride in those jobs) predictably leads to demoralization, demotivation and a cascading path down a whirlpool of failure.

The disadvantages of the system for day to day operations is well demonstrated by the issues around the ordering system. It is stated to be overly complex and requires a large deal of repetitive information to be input for multiple orders on one sample, plus specialist data entry knowledge that requires every joint order to have exactly the same timestamp. Ordering was the first accession where staff recognised that information is sometimes sent to the wrong staff, both arriving where it shouldn’t and not arriving where it should.

... Further mis-processing is seen with the cancellation of orders when a patient is transferred to a hospital ward from the ED. The results of orders, particularly radiology, often need to be checked by senior staff, but the system has no functionality to enable efficient processing of orders that have normal results, and thereby require no further attention.

I do recall another EHR system, by the same vendor I believe, where an "upgrade" in the recent past led to orders ending up in the wrong places (link):

... Computers at a major Midwest hospital chain went awry on June 29, posting some doctors’ orders to the wrong medical charts in a few cases and possibly putting patients in harm’s way.

The digital records system “would switch to another patient record without the user directing it to do so,” said Stephen Shivinsky, vice-president for corporate communications at Trinity Health System. Trinity operates 46 hospitals, most in Michigan, Iowa and Ohio.

[In other words, data entered by clinicians was going into the wrong charts. How many charts were involved? Does the hospital system even know, I wonder? - ed.]

Less than two weeks later, an unrelated glitch caused Trinity to shut down its $400 million system for four hours at 10 hospitals in the network because electronic pharmacy orders weren’t being delivered to nurses for dispensing to patients, he said.

Not to pick on this vendor; these "glitches" seem to be occurring in many HIT vendor's products.

All this, dear readers, is simply madness.

... Patient record retrieval is an important aspect of the staff work with patients, therefore its efficiency and accuracy is of vital importance to the activities of the ED. Staff were particularly pointed about the deterioration of this functionality in FirstNet compared to the previous system EDIS. [ED information system - ed.] There were cases where records could not be found, confusion about where data was stored in the patient record with different staff writing the same information into different parts of the record, and the [manual] rewriting of records [requiring a large amount of additional labor and time, not exactly a commodity in an ED -ed.] due to insertion of content into the wrong record.

These are critical issues. If you can't get information out of a computer in a timely fashion, what you have is a very expensive doorstop. I also note that a document imaging system that images hand written charts would not have these problems...

Prof. Patrick addresses the oft-heard canard that such complaints are the complaints of "Luddite doctors", old dogs who simply don't want to learn new tricks:

The staff in the ED are now generally experienced at using some form of clinical information system, many for over 10 years. This experience gives them a keen sense of what is possible with technology as well as the deficiencies in the existing systems. Combining this experience and knowledge with a sense of professional responsibility for process improvement enables them to judge quite acutely when a system is well designed or not. Hence their observations about elements of systems that are not parsimonious enough for optimal clinical efficiency deserve to be respected.

That it even needs to be written that the opinions of experienced medical professionals on the tools they are coerced to use by non-medical outsiders "deserve to be respected" gives testimony to my observation of a cross-disciplinary invasion of healthcare by the IT profession (among others).

Workflow and dataflow and the continuity of these processes are vital to the smooth running of a complex socio-technical process. ED staff have shaped these flows over a period of years and socialised all staff into the streaming. The directors have found that with the workflow of staff needing to use both clinical and nursing notes at the same time, their separation in FirstNet is deleterious. One department considered that the many nursing and medical notes accumulated over a day had to be kept in a single continuous sequence in the clinical record. Their workaround was to keep the one note page open for 24 hours to maintain the needed continuity in the patient record and avoid staff using a significant amount of time at the computer searching for needed information.

Nemeth and Cook explain how an ED EHR can be developed and marketed that interferes with, not supports, the workflows common in ED"s worldwide, in "Hiding in plain sight: What Koppel et al. tell us about healthcare IT", J Biomed Inform. 2005 Aug;38(4):262-3:

... On the surface, healthcare work seems to flow smoothly. That is because the clinicians who provide healthcare service make it so. Just beneath the apparently smooth-running operations is a complex, poorly bounded, conflicted, highly variable, uncertain, and high-tempo work domain. The technical work that clinicians perform resolves these complex and conflicting elements into a productive work domain. Occasional visitors to this setting [i.e., IT personnel, non-medical bureaucrats, etc. - ed.] see the smooth surface that clinicians have created and remain unaware of the conflicts that lie beneath it.

The technical work that clinicians perform is hiding in plain sight. [Hiding from the uninformed, that is - ed.] Those who know how to do research in this domain can see through the smooth surface and understand its complex and challenging reality. Occasional visitors cannot fathom this demanding work, much less create IT systems to support it. Progress in healthcare IT systems relies on scientific data on the actual, not the perceived, nature of day-today operations.

It increasingly seems the only place the faux-expert, healthcare-facilitor, cybernetic snake oil salespeople are going ot learn this simple lesson is in the courtroom...

A recent article in the press has presented evidence that access block times in EDs across the state of NSW are worsening.

As regards causality, it perhaps takes being an IS dept. director in a hospital - or holding elected office - to be unable to recognize the nose at the front of one's face.

Prof. Patrick wraps up this section of his forensic analysis as follows:

A number of conclusions can be drawn from the study:

1. Staff are entirely dissatisfied with the SBB and they feel that the deliverables have significantly failed to match the promises.

2. The Directors see that the HSS have failed in their support of the frontline of emergency care across the Sydney basin and their practices are decidedly lacking in proper engagement with the user community which should be their primary concern.

3. Some of the consequences of the HSS decision not to provide the reports needed by the Directors have lead to them being seriously hampered in being able to monitor the quality of their own department’s practices and wider changes and trends in community health.

4. The inefficiencies introduced by this technology have lead to a litany of complaints about its behaviour that have gone unheeded over the past three years.

5. It has lead to major strategies to work around the system by staff at all levels, to the point of complete avoidance by some staff.

The major consequences of these failings in the eyes of the ED directors are:

• Lost productivity and inefficiencies,
• Increased risks to patients,
• Disillusionment of staff and loss of morale.

Considering my own relative's injuries originating in an ED EHR mishap, I think I can safely add to Prof. Patrick's final tally an additional item:

• Patients have been harmed.
  

This is not a pretty picture.

I'm confident the Australian legal system abhors negligence as much as our own here in the U.S. If patients are injured and/or die as a result, considering that the Programme leadership and IT vendors knew - or should have known - of these deficits, it would not surprise me if criminal negligence charges begin to appear.

These issues are not exactly rocket science, and an expanding literature base has been appearing in recent years. See, for example, my recent post "An Updated Reading List on Health IT" at this link.

I can only add that my own relative was nearly killed as a result of a number of the phenomena described in Prof. Patrick's analysis.

More on other sections of his report at another time.

-- SS

March 6, 2011 addendum:

Also see my new post "What to do about the state of the ED EHR in NSW?"

-- SS

Mar. 8, 2011 addendum:

Prof. Patrick has added a new section to his report, entitled "The Future Pathways for e-Health in NSW." It is available at this link (PDF).

It inoculates against most of the 'Ten Plagues' that bedevil health IT projects (such as the IT-clinical leadership inversion, magical thinking about the technology, and lack of accountability):

More on the Pathways at my post here.

The de facto "National Program for IT in the HHS" here in the United States needs a similar inoculation.

--SS