BLOGSCAN: Forensic Statistics

Several interesting points are raised in the newsletter of the American Association of Physicians and Surgeons (AAPS) in a post entitled "Forensic Statistics" in their July 2011 newsletter headlined "Numbers." Healthcare Renewal is cited:

Forensic Statistics

While claims from RCTs fail to replicate about 20% of the time, the problem with epidemiology is so bad as to constitute a crisis, writes S. Stanley Young (“Everything Is Dangerous: a Controversy,” National Institute of Statistical Sciences, June 2008, www.niss.org). Fewer than 20% of nonrandomized trials [e.g., observational studies - ed.] replicate; i.e. 80%-90% of epidemiologists’ claims are false.

More than $1 billion in grant/tax money flows to institutions with reproducibility problems, Young states. A fundamental flaw in their methodology is to ask multiple, often hundreds to thousands of questions, of the same data set. It’s like playing “maverick solitaire”: given 25 randomly selected cards from a deck of 52 playing cards, the probability of being able to arrange them into 5 “pat hands” (e.g. a full house) is 98%.

Since data miners are good at concealing their footsteps, critics need full access to the raw data and the code used for the statistical analysis—often not forthcoming.

The EHR software that is supposed to support all this “research” and to guide medical treatment also needs a forensic evaluation, writes Scot Silverstein, M.D., of Drexel University (see http://hcrenewal.blogspot.com). He cites such an evaluation of the Cerner FirstNet system used in New South Wales, Australia, done by Prof. Jon Patrick. The authoritarian implementation processes of the governmental HIT “support” staff were familiar to Silverstein, such as disenfranchising the clinical staff and failing to acknowledge the validity of complaints.

“Healthcare reform” demands acceptance because it claims to be based on science. But then, so did Communism.

The "maverick solitaire" data mining issues (to be used, no doubt, in future "comparative effectiveness research" based on EHR data) are a additional concern to those I raised in an essay "The Syndrome of Inappropriate Overconfidence in Computing: An Invasion of Medicine by the Information Technology Industry?" in the AAPS journal several years ago (PDF).

I can also add: where are the RCT's of EHR/CPOE systems?

-- SS

More on Perversity in the Healthcare IT World: Is Meditech Employing Sockpuppets?

(Note to readers: also see my Jan. 7, 2010 followup post "Socky the Meditech Sockpuppet on Vacation?")

At "Are Dissmissive Industry and Government Reactions to Physician Concerns about EHR's and other Clinical IT Simply Perverse?" I observed that cavalier dismissals of physician reports on HIT unusability and difficulties fit quite well the definition of "perverse:"

Merriam-Webster dictionary:

Perverse (adj).
Etymology: Middle English, from Anglo-French purvers, pervers, from Latin perversus, from past participle of pervertere
Date: 14th century

1 a : turned away from what is right or good : corrupt b : improper, incorrect c : contrary to the evidence or the direction of the judge on a point of law
2 a : obstinate in opposing what is right, reasonable, or accepted : wrongheaded b : arising from or indicative of stubbornness or obstinacy
3 : marked by peevishness or petulance : cranky
4 : marked by perversion : perverted

In a later post, "An Honest Physician Survey on EHR's" I reported on the comments submitted by hundreds of physician members of the American Association of Physicians and Surgeons (an organization mainly of physicians in private practice who strongly support physician independence and other conservative views, founded in 1943) in a 2008 survey about HIT:

AAPS - PHYSICIAN ATTITUDES & ADOPTION OF HEALTH INFORMATION TECHNOLOGY (PDF)

The common theme in their feedback was how HIT in its present form disrupted private practice physicians, distracted them from the physician-patient relationship and impaired their ability to properly care for patients. See the above-linked post and AAPS survey report.

An anonymous, usually lively and even combative reader "IT Guy," who claims to be an IT professional at an HIT vendor, on occasion leaves comments to my HIT posts.

These are typically in the form of unsubstantiated refutations of the material in the posts, and ad hominem attacks in the unmitigated defense of HIT, e.g., referring to this writer as "a teaching professor at a major university who has virtually no understanding of statistical analysis" or as a "grandstanding self-promoter" (See, for example, here at January 5, 2010 8:53:00 AM EST. Read the entire thread.)

In the latest case, "IT Guy" commented on my report of the AAPS HIT survey responses as follows:

IT Guy said...
It's the March of the Ludites.

Thanks for a very funny post.

January 4, 2010 12:28:00 PM EST

I failed to see the humor in dozens of adverse comments about HIT from private practice physicians, and replied with a link to my initial post about HIT industry perversity mentioned above, which elicited the even more perverse response:

IT Guy said...
No one is dissmissive of legitimate concerns. Luddites are a different story. Most of the "concerns" in that diatribe are of the Luddite variety.

January 4, 2010 1:33:00 PM EST

In other words, a survey of physician concerns is a "diatribe" and it is up to the "IT guys" to determine which physician concerns are "legitimate" and which are of the "Luddite" variety.

"IT Guy" remains anonymous and has been so since he first started posting comments here, despite prodding to reveal his identity to better facilitate an understanding of where his/her viewpoints arose. He/she has neglected to do so.

Even the blogger profile is blank, click to enlarge:




Now, I welcome anonymous comments and have a thick skin - to a point. When the comments go ad hominem or perverse, I do consider deleting them.

However, when such comments are potentially revelatory of major issues, I promote and amplify them - as now. Read on.

This person also apparently uses the anonymous moniker "Programmer" at the HIStalk blog where he similarly attacks my comments made under my actual name S Silverstein or under MedInformaticsMD. The HIStalk site owner actually edited out defamatory comments made about me in Oct. 2009 at HIStalk comment #28 at this HIStalk comment thread and apologized for this entry on his blog:

#28 Programmer [at HIStalk blog - ed.]October 20th, 2009 at 11:57 am

Yes, it’s that simple. If you select for pre-IT and post-IT data and use a large enough sample size the other factors with equal out. If the sample size is large enough you should have a relatively small margin of error.

[i.e., "Programmer" -- who I soon show is also "IT Guy" -- opines that in comparing clinical adverse event rates pre- and post healthcare IT installation, all you need is a large enough sample size, which then nullifies or cancels out, for example, changes occurring over time that are not related to the intervention, and other potential confounders in a pre-post comparison. If only evaluation studies in healthcare informatics were that simple ... it is concerning that IT vendor personnel might have such beliefs - ed.]And the fact that a teaching professor at a major university has virtually no understanding of statistical analysis makes me say “at least I don’t have to worry about losing my job to one of his students."

[Latter sentence was removed by HIStalk owner - ed.]

Now, back at HC Renewal see this combative comment thread where "IT Guy" a.k.a. "Programmer" refers to that removal, and repeats the above statistical faux pas and ad hominem ("just to make sure I read the whole thing"), and adds another ad hominem for good measure. I let them remain. (Comment dated October 20, 2009 1:35:00 PM EDT.) HIStalk's "Programmer" and HC Renewal's "IT guy" are apparently one and the same.

Getting to the core of this posting, I repeat, when such comments are potentially revelatory of major issues in HIT, I promote them - as here.

The raison d'être for this posting is an interesting pattern:

Before "IT Guy" posts comments at HC Renewal, "hits" appear from a major health IT vendor's IP in our publicly-accessible Sitemeter log, with outclicks to the comment sections of posts where "IT Guy's" comments then appear.

For example:

Domain Name (Unknown)
IP Address 12.11.157.# (Medical Information Technology) [Meditech - ed.]
ISP AT&T WorldNet Services
Location
Continent : North America
Country : United States (Facts)
State : Massachusetts
City : Milford
Lat/Long : 42.1544, -71.521 (Map)
Language English (U.S.)
en-us
Operating System Microsoft WinXP
Browser Internet Explorer 6.0
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)
Javascript version 1.3
Monitor
Resolution : 1024 x 768
Color Depth : 32 bits
Time of Visit Jan 4 2010 11:49:49 am
Last Page View Jan 4 2010 12:55:12 pm
Visit Length 1 hour 5 minutes 23 seconds
Page Views 12
Referring URL
Visit Entry Page http://hcrenewal.blogspot.com/
Visit Exit Page http://hcrenewal.blogspot.com/
Out Click 0 comments
https://www.blogger.com/comment.g?blogID=9551150&postID=4799128165855153590&isPopup=true
Time Zone UTC-5:00
Visitor's Time Jan 4 2010 11:49:49 am
Visit Number 643,748

At the time of the outclick, there were "0 comments" to that post, as shown in the log above. Shortly after, IT Guy's aforementioned "Funny March of the Luddites" comment appeared ... as comment #1.

Likewise today, several "hits" appeared from IP 12.11.157.# with outlinks to the comment thread, for instance as seen below when only 23 comments were present, mostly from "IT Guy", Dr. Poses and myself:

Domain Name (Unknown)
IP Address 12.11.157.# (Medical Information Technology)
ISP AT&T WorldNet Services
Location
Continent : North America
Country : United States (Facts)
State : Massachusetts
City : Milford
Lat/Long : 42.1544, -71.521 (Map)
Language English (U.S.)
en-us
Operating System Microsoft WinXP
Browser Internet Explorer 6.0
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)
Javascript version 1.3
Monitor
Resolution : 1024 x 768
Color Depth : 32 bits
Time of Visit Jan 5 2010 9:38:07 am
Last Page View Jan 5 2010 9:57:40 am
Visit Length 19 minutes 33 seconds
Page Views 3
Referring URL
Visit Entry Page http://hcrenewal.blogspot.com/
Visit Exit Page http://hcrenewal.blogspot.com/
Out Click 23 comments
https://www.blogger.com/comment.g?blogID=9551150&postID=4799128165855153590&isPopup=true
Time Zone UTC-5:00
Visitor's Time Jan 5 2010 9:38:07 am
Visit Number 644,308

After that, more comments from "IT Guy" appeared starting with ... #24.

The pattern has remained consistent.

Now, the evidence is circumstantial but it does not take a Sherlock Holmes to realize it is quite likely this commenter is an employee of a healthcare IT vendor named in the above links, Medical Information Technology, Inc., a.k.a. Meditech.

I am concerned that a possible employee of an HIT company - any HIT company -- might find physician concerns about HIT as serious as those expressed in the AAPS survey "funny" and "of the Luddite variety." I also am concerned that an employee might think that in situ pre-post evaluations of the technology need not take into account possible confounders.

If this person is an HIT vendor employee and IT professional at this HIT company or any other -- I think it likely he/she holds such a position at some HIT company and such attitudes -- then a number of questions are raised:

• How common is this attitude among HIT vendor employees? Is this a systemic problem?

• How do such attitudes translate into satisfying customer requests for remediation of HIT defects and problems?

• Should HIT vendors be doing better due diligence in their hiring practices to assure they hire IT personnel with a service mentality and who understand that clinicians are the enablers of medicine, they the facilitators? (A point my graduate healthcare informatics students are taught and grasp readily.)

At the very least, perhaps employees of HIT companies (such as the one in the logs above at Meditech, whoever they may be) should pay more attention to improving HIT, rather than spending 1 hour 5 minutes 23 seconds reading 12 posts here during business hours.

"IT Guy" is welcome to continue submitting anonymous comments, but if they contain ad hominem they will be deleted.

-- SS

Addendum 1/5/10:

A HC Renewal reader with an MBA non-anonymously relates the following (emphases mine):

In reading this thread of comments I have to believe IT Guy is a salesperson. My only question is: Were you assigned this blog or did you choose it? We had this problem a number of years ago where a salesperson was assigned a number of blogs with the intent of using up valuable time in trying to discredit the postings.

In my very first sales class we learned to focus on irrelevant points, constantly shift the discussion, and generally try to distract criticism. I would say that HCR is creating heat for IT Guy’s employer and the industry in general.

I find it sad that a company would allow an employee to attack anyone in an open forum. IT Guy needs to check with his superiors to find out if they approve of this use of his time, and I hope he is not using a company computer, unless once again this attack is company sanctioned.

Steve Lucas

I think that is an interesting possibility - someone paid to disrupt. It fits, and again invoking Sherlock Holmes, there is the means, the motive and the opportunity. Time for another definition:

Sock puppeting: "the act of creating a fake online identity to praise, defend or create the illusion of support for one’s self, allies or company." (NY Times)

If true (unfortunately for the salesperson), I make this observation:

To most of the readers of Healthcare Renewal, who find a focus on irrelevancy and irrationality to be signs of foolishness and hysteria (we clinicians have seen it all, by the way), this salesperson has nothing to sell.

Another perversity also comes to mind. If what Mr. Lucas suggests is indeed occurring, a company behind such actions would be exhibiting self destructive behavior in trying to disrupt and discredit those who could actually help them to make better products and be more competitive. I remind that patients are the true "customer."

It also follows that, if this analysis is true, the defamatory attacks left at HIStalk and here at HC Renewal would have been made with foreknowledge of their falsity and with malice.  I have informed the company's General Counsel of this information and hope that will be sufficient to cause the harassment from their company to cease.

Defamation, I also add, is certainly not a particularly wise HIT vendor strategy with the HIT industry under investigation by US Senator Grassley (see Oct. 2009 PDF letter to a number of vendors and management consultant firms here).

Having worked in pharma, however, another self-destructive industry due to its internal pathologies, I've seen worse done to critics. Incidentally, another probable blog troll/sockpuppet comment from that industry is in the comments section at this Jan. 2008 post.

-- SS

An Honest Physician Survey on EHR's

I often believe surveys of physicians about EHR's do not present the results candidly, but rather are selective in what is reported - and what is omitted - and generally sugar-coated. Examples of very candid reports are rare, probably due to pushback, such as this recent report on ED EHR's in New South Wales from Australia.

Often the pushback takes the form of the report "lacking peer review", which in a non-free market, vendor-dominated situation (as in pharma, with money flowing everywhere but up) is as likely to produce censorship or, at best, groupthink, as objective science.

Here is one that is candid, by the American Association of Physicians and Surgeons.

The Association of American Physicians and Surgeons , founded in 1943, regularly testifies before the U.S. Department of Health and Human Services regarding development and implementation of health information technology. It consists largely of physicians in private practice. I've been a member of the organization, but was not at the time this survey was performed and written up:

PHYSICIAN ATTITUDES & ADOPTION OF HEALTH INFORMATION TECHNOLOGY (PDF)

Results Compiled on 6/9/2008

Specialties Responding:

Family Practice 73
Psychology 38
Internal Medicine 33
OBG 27
Orthopedic Surgery 27
Ophthalmology 26
General Surgery 22
Dermatology 21
ENT/OTO 15
AN 14
Neurology 13
Pain Management 13
Urology 12
Pulmonary
Diseases 11
Neurosurgery 8
Vascular Surgery 6
Cardiology 5
Radiology 5
Gastro 5
Emergency Med 4

I am simply reproducing some of the comments received below without additional comment:

A patient's medical history is nobody's business but the doctor's and the patient's.

All EHRs examined are cumbersome and ineffiecient

As a 'computer programmer,' can see pushing buttons to make statements about a patient's health, really makes patient care more distant, takes the personal, hand-touched art out of practicing medicine, AND lends itself to inaccuracies and errors

As a primary care physician, I rarely see patients for one problem, yet most EHRs Ive tested are based on the 'problem/visit' models.Expanding the visit to include the 'oh, by the way, doc's' is cumbersome and even more time consuming.

Better--paper record (for patient, also, to keep)

Big Brother is watching you--1984

Can't view my study printouts and look for change--pages 'turn' too slow

Comment: I do write notes on my computer but it is not part of any 'system.' I do not send bills via computer.

Comment: as anesthesiologists, we use the hospital's EMR, but we haven't implemented our own. Possibility that it won't lead to improvements in quality of care

Comment: However, I work with a physician's group to promote EHRs and run into many obstacles

Comment: we have spent upwards of $200,000 on Nextgen software plus hardware for our clinic and have never been able to make it function over the past 5 years

Compatibility

Concern about presumed access to record by multiple non-insurance third parties.

continued cost of support, maintenance, and updates of hardware and software

cost benefit ratio too high

Cost of upgrades

degradation of personal dr/pt relationship. Instead of a conversation between two people there is the intrusion of a mechanical 'other.'

Distraction from personal patient care

diverts attention from patient to data processing

Doesn't work. Studies show no better. Push for EHR due to 'Big Brother's' appetite for info and control.

Don't need it or want it. Concern about accuracy. Many of the automated consult letters I receive contain glaring errors and omissions.

EHR generates false pre-programmed info that does not truly reflect the time actually spent with the patient allowing the MD to 'upcode' for the visit and bill higher. It is more honest for me t spend 20' with my patient and write wo words of actually pertinent info.

Ehr in use for nurses only at my second site and it slows down the care they give.

EHR is most impersonal. It does not give a fell for what is going on with patient.

EHR notes are poor, very poor. Full of useless verbiage and usually no place for physicians to add specific notes (or they are lost in the mass of irrelevant detail automatically supplied by the program. Also encourages physicians, who are often pressed for time, to make any specific notes.)

EMR are very time consuming, result in production of lengthy repetitive notes of questionable clinical value and reliability.

Comment: companies go out of business and new systems need to be installed.

Comment: Federal and state govt. will continually add requirements

EMR printouts contain extensive boiler plate data. The real data is hard to glean from the chaff.

Fills the chart with negative (non-used) information

Getting the computer 'right' will become more important than taking care of the patient.

Have started process

Have used EHR and find written records more reliable and practical

I am a fulltime ER doctor. I have no say but if they go electronic, I go.

I am blind

I am concerned with control that's being exercised here. There's no room for creativity, judgement and financial shortfalls. If the government or insurance companies would take the overhead including this financial then it might be palatable

I am not a good 'typer'--on a keyboard--I do not type at all. Don't want to type, never will. I am not trained as a secretary or clerk.

I do not want to have to turn on a computer everytime I speak to a patient or need a chart with consulting with another physician or a pharmacist.

I feel like I would be a secretary to enter data on my patients so that government can easily slide into socialized medicine.

I hate typing and anything that distracts me from writing and examining

I have not found a system that will speed up my patient encounter. All make it slower--with keyboard--not patient--time.

I see no benefits; would certainly disrupt my thinking process.

If mandated--no standard for format. Took 15 years to finally get standard for electronic billing.

If purchased would be faced with frequent expensive changes to format. They still can't get the new NPI number to work! Everyone I know who spent $50,000 to buy a system either junked it or are planning to!

In 2000, I lost my billing staff. Led to a computer-based billing system due to that. The transition was horrible! I could not use the system myself and training for staff was expensive; the IT guy was expensive; billing personnel who had experience with my system were few. In the end, 4 years later, I had an AR $136,000 and as aconsequence, I closed that practice

Inappropriate EMR causes defocus from reason for visit, etc. Problem with sketches.

Still would need 2 charts--one paper, photos, etc. and 1 EMR

Inefficient. They do not provide the clinical data that I need.

It is impossible to skim through an electronic record to find data. It is impossible to sketch the affected anatomy in electronic records. If the computer breaks down or the technology becomes obsolete, the patient record disappears. It takes too long to enter data into a computer.

You still need a paper chart to share reports and other patient paper records.

It will be of no value in my single practitioner spine surgery practice lack of any standard format/compatibility of various systems

Lack of personal patient interplay

Less time with patient, more time with computer. There are better ways to give ER docs access to patient's med records. Survey ER docs to learn what info they would need when pt. is unconscious. Put that into pt. ID card using 2D Barcodes or magnetic strips. Card readers in ERs can then access that info. If AAPS helped develop and sponsor this for its members, it could be a source of $$ for AAPS

loss of dr-patient relationship

Loss of patient control over privacy of records

Loss of quality of patient's personal records. Physical deterioration of data over many years

Inadequate accuracy of voice recognition technology Lack of evidence that EHRs
are any better or equal to paper records except in narrow applications

I purchased an EHR system and was unimpressed. Main reason: prefer personal notes. I believe dictated notes are more specific and detailed and are customized for each patient visit

May not be able to get to computer records in case of computer crash or power failure (eg. Katrina).

most software have major problems in functionality and changes how physicians practice in a potentially negative way

Must have voice recognition for input at 100% accuracy and reliability.

my patients are given copies of all reports (lab, x-ray, consults) as they are collected and told to keep in their medical file

No adequate voice recognition systems

No clinical evidence that this improves outcome. No clinical evidence that there is a return of investment.

No evidence that EHR will improve care or reduce costs to the patient/doctor/healthcare system.

No improvement in quality of care provided.

Once they force us into the more expensive, time-consuming system that does not work, they
own us! It is too easy for the courts, government, hackers and insurance companies to take 'all,' once it is in the system!

One of worst business decision we made.

open source software is available (but VA Vista split into Open Vista and World Vista groups and is written in a language that is not known to many programmers) but I haven't taken the time to find something that could work--I don't know if any of them can keep up with the government requirement

oppose all government interference

Paper charts are much more accurate and efficient for me.

Patient safety

perpetuation of errors

Preoccupation with the computer takes time from patient. Increased errors from EMR especially CPOE. We already have well established safety checks and reviews in our system for tracking tests and medicines. No system especially CPOE have been tested for safety and efficacy nor approved by any regulatory agency and thus the alteration of care from these (?) is nothing but an experiment and patients have not signed consent. Preoccupation with the computer takes time from patient. Increased errors from EMR. We already have well-established safety checks in our system for tracking tests and medicines. No systems have been tested for safety and efficicacy nor approved by any regulatory agency

Reduced time with patients. My patients complain about other doctors playing with EHR computer instead of looking at them during visit.

reliance on psychological pen and paper tests

Slower system than handwritten notes

slows review of chart at each office visit

some parties are paying $10,000 per month for technical support

Sorry, I cannot fill this out--I have visual problems

Still building the software

Studies are not showing conclusive evidence that EMRs improve patient care or safety, but do increase practice costs.

systems are difficult to implement; I've been trying for 2 years

The EHR in the hospital slows me down. A paper record is more efficient for me.

The systems seem to impede quality clinical care and passing along of relevant clinical information

There is not one advantage to me, at all!

There is one product I would use, PRAXIS. [www.informed.com] I would need $30,000 infrastructure and $30,000 adoption overhead grant in order to do so

They don't improve patient care--just adds to overload

time taken up up for data; focus on computer rather than on patient in the exam room

Too rigid. I like to draw pictures of what I see on ophthalmological exam.

Typed, dictated note can be read much more quickly. I use a print about 1/2 the size of your print on this page and there are perhaps 4 or 5 pages of regular print per page which I read without glasses. Computer and power problems do not hide my records.

Unfunded mandate with huge cost in a severely declining reimbursement arena.

without a personalized note, it is worthless.

Read the entire report as linked above. No additional comments are needed.

-- SS