What If the Institute of Medicine Wrote a Report and Nobody Followed it? - the Case of the Standards for Developing Trustworthy Guidelines

For over 20 years, clinical practice guidelines (CPGs) have been touted to improve health care quality and control costs.  Enormous numbers of guidelines have been developed, but with seemingly little impact on health outcomes.  While some of those leading health care organizations have predictably blamed individual practitioners for obstinately ignoring or challenging guidelines, there is increasing evidence that maybe the guidelines themselves are part of the problem.

 An Example of a Guideline that Apparently was Not Trusted

One example Dr Wally Smith and I have taught in our recurring mini-course on why physicians fail to follow guidelines (and otherwise appear not to practice in accord with others' wishes) is that of the guidelines on management of depression in primary care.  Most existing guidelines urge physicians to screen patients for (presumably mild-to-moderate) depression and treat them aggressively, with emphasis on the use of the newer anti-depressants.  These guidelines, in turn, were based on numerous published randomized clinical trials that showed that these drugs were safe and efficacious.  Yet multiple studies showed that physicians failed to follow these guidelines, and various attempts to improve their adherence did little.  So for years the assumption was that physicians at best experienced practical and system barriers to follow these guidelines, and at worst were ill-informed or irrational. 


However, information that came out gradually during the early part of the 21st century suggested that perhaps the problem was within the guidelines, not the health care professionals.  First, documents produced during New York Attorney General Eliot Spitzer's lawsuit against GlaxoSmithKline about the marketing of one of these drugs (Paxil, paroxetine) suggested that the company had suppressed clinical trial data that reflected poorly on the drug (See Kondro W.  Drug company experts advised staff to withold data about SSRI use in children.  Can Med Assoc J 2004; 170: 783.  Link here.)    These suspicions were later fleshed out  by consideration of documents further disclosed in litigation (e.g., see this post and its links).  Then several studies, most particularly that by Erick Turner and colleagues, showed that numerous trials of new anti-depressants had been suppressed, that is, never published (Turner et al. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008; 358:252-260.  Link here).  When the results of these trials were added to those that were published, the efficacy of anti-depressants was no longer so clear.  So maybe the guidelines that physicians did not follow were not trustworthy, and should not have been followed in the first place.

Would IOM Standards to Improve Guideline Trustworthiness Help?

So in 2011, the prestigious Institute of Medicine released a report on the development of  better standards to produce more trustworthy guidelines (Clinical Practice Guidelines We Can Trust.  Link here.)  We posted about that report here, but noted that it was receiving little other attention, an example of the anechoic effect. 

A few weeks ago, an article appeared documenting a study meant to assess the the trustworthiness of clinical practice guidelines published soon after the IOM report.  Its title telegraphs the results. ( Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet Institute of Medicine standards: two more decades of little, if any progress.  Arch Intern Med 2012.  Link here.)

Methods and Results

The investigators selected a random sample of 114 individual guidelines available during June, 2011 stratified by 26 clinical topics. The versions of the guidelines used were those archived in the National Guideline Clearinghouse (NGC) maintained by the Agency for Healthcare Research and Quality (AHRQ).


The goal of the study was to "examine adherence to the IOM standards" by guidelines published after the standards were published.  Actually, the study only assessed adherence to 18 of the 25 standards espoused by the IOM (because the remaining seven were "too vague and subjective to be analyzed.")

Furthermore, the criteria used to determine if a specific guideline met each of the three items above were rather lax:

In evaluating each guideline summary, care was taken to be as liberal as possible in considering that a standard was met when the individual guideline summary provided any information pertaining to that particular standard.

Nevertheless, using these lax standards to only evaluate adherence to 18/25 guidelines, the authors found that "the overall median number of IOM standards satisfied (out of 18) was 8 (44.4%) ....  Fewer than half of the guidelines surveyed met more than 50% of the IOM standards."

An examination of the details of the study's methods reveals things are even worse than that.

Analyzing the Study's Methods to Find that Things Are Worse Than They Seem

Review of the study's methods show that they provided a very optimistic view of adherence to the IOM standards.  As noted above, the study did not look for adherence to all of the IOM standards.  Moreover, those they did consider were simplified and made less rigorous.  For example, Standard 2 from the IOM on management of conflict of interest (COI) was:

  STANDARD 2
Management of conflict of interest (COI)

2.1
Prior to selection of the Guideline Development Group (GDG), individuals being considered for membership should declare all interests and activities potentially resulting in COI with development group activity, by written disclosure to those convening the GDG.
- Disclosure should reflect all current and planned commercial (including services from which a clinician derives a substantial proportion of income), non-commercial, intellectual, institutional, and patient/public activities pertinent to the potential scope of the CPG.

2.2
Disclosure of COIs within GDG
- All COI of each GDG member should be reported and discussed by the prospective development group prior to the onset of their work.
- Each panel member should explain how their • COI could influence the CPG development process or specific recommendations.

2.3
Divestment
- Members of the GDG should divest themselves of financial investments they or their family members have in, and not participate in marketing activities or advisory boards of, entities whose interests could be affected by CPG recommendations.

2.4
Exclusions
- Whenever possible GDG members should not have COI.
- In some circumstances, a GDG may not be able to perform its work without members who have COIs, such as relevant clinical specialists who receive a substantial portion of their incomes from services pertinent to the CPG.
- Members with COIs should represent not more than a minority of the GDG.
- The chair or co-chairs should not be a person(s) with COI.
- Funders should have no role in CPG development.

However, the study boiled all this down to three items:
-  COIs stated
-  Chair has COI
-  Co-chairperson has COI

Thus the study did not address standards requiring full and complete disclosure (not just some disclosure) of all COIs; consideration of how the COIs might influence the particular guideline; divestment of specific types of conflicts of interest, that is, financial investments, and cessation of participation in marketing activities or advisor boards; minimization of conflicts of all members of the committee; and barring of participation of funders in guideline development.

Even so, the guidelines assessed did a very poor job upholding even these few liberalized standards regarding conflicts of interest.  Of the guidelines assessed, less than half, 46.8% provided ANY disclosure of conflicts of interest.  Those written by sub-specialty societies were particularly opaque in this regard.  Less than one-third, 29.3%, provided any disclosure.  Thus the majority of guidelines assessed were not at all transparent about conflicts of interest affecting the guideline development process.

Furthermore, of the 46.8% of all the guidelines which made any disclosures of conflicts of interest, 71.4% admitted their chair people HAD a conflict of interest.  Thus, only (0.468 * [1 - .714]) = 13.3% provided assurance that they fulfilled the single requirement (from standard 2.4 above) that the chair person did not have a COI.  For the guidelines written by sub-specialty societies, by a similar calculation, only 12.2% provided an assurance that the chair had no COI.  (The proportions providing assurance that the co-chair people had no COI were even lower.)  Thus the vast majority of guidelines did not clearly follow two straight-forward standards for minimizing the effects of conflict of interest, that the guideline committee chair and co-chair should not have any relevant conflicts. 

Given the miserable results concerning even minimal adherence to some of the IOM report's conflict of interest standards, it is likely that almost no published guidelines from 2011 came close to fulfilling the full set of IOM standards.  Despite the best efforts of the IOM, it appears that guideline developers have not progressed at all towards providing trustworthy guidelines. 

Summary

An editorial (Shaneyfelt T. In guidelines we cannot trust.  Arch Intern Med 2012) accompanying the article by Kung and colleagues summarized its results thus:

The same problems that have plagued guideline development continue to plague guideline development; namely, their variable and opaque development methods, their often conflicted and limited panel composition, and their lack of significant external review by stakeholders throughout the development process.   As a result, the trustworthiness of guidelines is limited.

While guidelines may have seemed to be a promising method to improve health care in the early 1990s, they have failed to live up to that promise.  Shaneyfelt was not optimistic they would improve in the future:

I am not optimistic that much will improve.  No one seems interested in curtailing the out-of-control guideline industry.

On the other hand, in my humble opinion, it is not that on one is interested in better guidelines.  It would clearly be in the best interests of patients and the public, and of health care professionals who care about the quality of their practice and the outcomes of their patients to curtail that industry.  The issue is why patients', the public's, and professional's interests were ignored.

Neither Shaneyfelt nor Kung et al discussed why there has been so little attention to patients' and the public's health, and to health care professionalism in all this.  For a quick answer, we do not have to look far on Health Care Renewal. 


In fact, the IOM report on guideline development from 2011 was a serious challenge to the powers that be in health care.  In particular, it challenged the cozy relationships that had grown up among the organizations that undertook guideline development and the health care professionals on guideline panels on one hand and organizations that stand to gain were specific guidelines to favor their products, services, and agendas on the other.  The standards mandated transparency and honesty about conflicts of interest affecting guideline committees and the organizations which assembled them, and if upheld would have greatly reduced these relationships.

Now it turns out that the guideline standards have been honored mainly in the breach.  Of course, these standards, while increasing trustworthiness, would have cost a lot of medical societies considerable commercial funding, and would have cost a lot of health care professionals on guideline panels considerable personal wealth.  These standards would probably also have cost a lot of companies whose products and services were addressed by guidelines to lose revenue.  So it is not surprising that the IOM standards were ignored.  Their implementation would have cost too many people who are financially benefiting from the status quo too much money.  And these people, that is, leaders of professional societies dependent on commercial outside funding, health care professionals and academic used to financial support from commercial interests, and health care corporations are good at making sure their interests are not ignored, even if their interests conflict with those of patients, the public, and well-intentioned health care professionals.  

So, the flouting of the well reasoned IOM guideline standards adds one more reason for patients and the general public to distrust modern health care and all those who "deliver" it, even to distrust well-intentioned health care professionals who have not been able to distinguish themselves from their colleagues who are too happy to help commercial interests while taking commercial money.  If health care professionals want to regain the public's trust, they could do worse than publicly declaring their intention to show that their practice in the future will be guided by trustworthy guidelines based on clinical research evidence and knowledge of biomedical science, drawn up by health care professionals independent of commercial interests.