How Manipulated Clinical Evidence Could Distort Guidelines - the Case of Statins for Primary Prevention

The new American College of Cardiology (ACC)/ American Heart Association (AHA) guidelines on the primary prevention of cardiovascular disease, which we discussed here, continue to generate controversy. 

Articles in the media and online-first in medical journals underscored some of the issues we discussed before.  Jeanne Lenzer, in a news article in the British Medical Journal, found that the chair of the guideline panel had important past conflicts of interest that were not previously disclosed.(1)  The chair denied their significance (look here).   Guideline panel members continued to justify their efforts, but in my humble opinion, raised no new evidence or logic to support it (look here  and here)

Bigger Questions about the Validity of the Clinical Evidence

However, several new articles suggested the need for a deeper look at one particular aspect of this case, the validity of the evidence from clinical research about the benefits and harms of statins for primary prevention.

JAMA published a summary of the Cochrane review that provided a basis for the guideline developers' confidence in the worth of statin therapy in primary prevention.(2)

results suggest that the benefits of statin therapy outweigh serious life-threatening hazards.

However, almost as an aside, it noted,

Some trials included participants with CVD, but rather than exclude these trials, we included trials that contained 10% or fewer participants with documented CVD.

Primary prevention in this case is defined as prevention for patients without existing cardiovascular disease.  There is evidence that statins may well have benefits that outweigh harms when given to patients with known cardiovascular disease, particularly documented coronary artery disease (CAD).  Mixing such patients in any numbers into "primary prevention trials" would likely exaggerate the benefits of statins.  Yet such not quite primary prevention studies were included in a systematic review of primary prevention. 

In addition, a commentary by one of the guideline developers defending the group's work also underscored the fact that many of the supposedly primary prevention trials they used as evidence were not pure primary prevention trials.(3)

Notably, the 2013 cholesterol guideline cut points were derived from the placebo rates for myocardial infarction, stroke, and cardiovascular disease death observed in the 3 exclusively primary prevention statin trials, Air Force/Texas Coronary Atherosclerosis Prevention Study (AFCAPS/TexCAPS), Management of Elevated Cholesterol in the Primary Prevention Group of Adult Japanese (MEGA) study, and the Justification for the Use of Statins in Primary Prevention: an Intervention Trial Evaluating Rosuvastatin (JUPITER).[4-6]

The wording again suggests that all the other trials used as evidence about primary prevention were NOT "exclusively primary prevention studies," and hence, as I noted above, may have been biased so that they would likely exaggerate the apparent benefits of statins in primary prevention.

This suggested to me that the systematic review that provided a basis for the guidelines' aggressive recommendations about statins for primary prevention, and the trials on which it was based needed further skeptical, critical reviews.

The Cochrane Review: Was Evidence about Statin Benefits vs Harms Manipulated?

We have frequently discussed the manipulation of clinical research.  By that we have meant design, implementation, analysis or dissemination of research in ways likely to further vested interests.  In particular, when drug, biotechnology and device companies sponsor and control clinical research on their own products, they may set up the research in ways likely to make their products look better than they actually are.  

We looked through the most recent Cochrane review of statins for primary prevention(7) with a skeptical eye out for suggestions that the trials reviewed could have been so manipulated.  We found quite a bit.

First, we found that not all trials reported on the "hard" outcomes that one needs to consider when evaluating benefit vs harm of statins in primary prevention.

Data on all-cause mortality were provided in 11 trials.[of 19, and hence missing in 8].

And,

 Twelve trials provided data on adverse events.[and hence 7 did not.]

 Also, data on specific adverse events was often not reported: myalgia and rhabdomyolysis were reported in only 9/19 trials; new onset diabetes mellitus in only 2/18; hemmorhagic stroke in only 2/19; abnormal liver tests in only 10/19; kidney dysfunction in only 4/19; arthritis in only 2/19, and by implication, cognitive dysfunction in 0/19.  

  Failure to look for all the possible bad outcomes of treatment could obviously bias the study in the direction of minimizing the harms of the treatment.

A substantial number of trials either failed to report on crucial aspects of their methods, or admitted to flaws that could have induced important biases.: 3/19 did not described randomization methods; 4/19 did not use double-blinding; 6/19 did not use intention to treat analysis; 7/19 did not report their drop-out rates.

So it is very surprising to me that the authors concluded,

In general there was low risk of bias ... though all trials were either fully or partially funded by pharmaceutical companies.

In my humble opinion, the Cochrane review showed many trials that had flaws could have biased their outcomes, and hence the outcomes of the overall review.  Some of the flaws clearly could have lead to biases that would have made statins look more efficacious, or less dangerous than they might actually be.  I do not understand the conclusion that the risk of bias was slow, and the lack of discussion about the direction the bias could have taken.

 Review of the "3 Exclusively Primary Prevention Statin Trials"

Given the Cochrane review's apparent lack of skepticism about methodologic problems in the industry funded statin prevention trials, I endeavored to take a closer look at the three trials that Dr Robinson held out as the real primary prevention trials.  Instances of manipulation, as we defined it above, for each trial are described below

AFCAPS/ TexCAPS(4)

Narrow Patient Population - This study excluded many patient for whom the statins were not contraindicated or warned against: uncontrolled hypertension; type 1 or type 2 diabetes mellitus on insulin or with a HgBA1C at least 10%; and body weight more than 50% "desirable limit for height."  (Based on the official contraindications and warnings for commonly used statins, e.g., see contraindications for Lipitor here, active liver disease, pregnancy for likely to become pregnant, nursing mothers, hypersensitivity to the medicine; and warnings: use of cyclosprine or strong CYP3A4 inhibitors, uncontrolled hypothyroidism, renal impairment.)  Thus the results may not generalize to many patients who would otherwise be considered statin candidates.  By excluding such patients, the results may bias the study towards minimizing the probabilities of harms that might occur were statins used on a wide population for primary prevention.

Unknown Randomization and Allocation Concealment Procedures - According to the Cochrane Review, the study report did not explain how randomization or allocation concealment were accomplished.  

Early Termination/ Multiple Comparisons - The study was terminated early based on an early look at the number of outcome events.  Two such early or interim analyses were planned.  Taking multiple looks at the data over time raises a multiple comparisons problem, and may lead to exaggerating the benefits of the treatment.(8). Furthermore, stopping early decreases the sample size and hence the power to find adverse effects of treatment.

Implausible Dropout Rate, Missing Data - According to the Cochrane Review, the study reported no dropouts.  This seems somewhat improbable, suggesting skepticism about the accuracy and completeness of the data collection.  On the other hand, a study chronology suggests that of 6605 patients who started the study, 6540 had data on complete endpoint status, suggesting missing data.  Since dropouts and missing data may be due to different reasons in different arms of the study, they threaten the validity of data about benefits and harms.  

Adverse Effects not Reported - The study provided no data about development of diabetes, hemmorhagic stroke, kidney dysfunction, arthritis, or cognitive dysfunction, suggesting incomplete data about harms, and hence bias towards minimizing harms.    

MEGA (5)

Narrow Patient Population -  [The patient population was not described in the main report, but only in an earlier methods article.](9)  The study excluded patients with congenital or rheumatic heart disease; chronic atrial fibrillation, current diagnosis of malignancy; poorly controlled hypertension or diabetes mellitus; current use of oral or parenteral corticosteroids; and other conditions at the discretion of the physician.  These exclusions seem unrelated to the contraindications or warnings on the stain label.  Again, such a narrow patient population reduces the generalizability of the study results, and may bias the study to minimizing the harms of statins.

Only Single Blind - This was an open-label study, so patients and physicians knew who got statins and who got placebo.  Such knowledge could have biased patient management, including how diligently particular diagnoses and outcomes were pursued, and biased data collected from patients or physicians.

Adverse Effects Not Reported - The study provided no data about diabetes, hemmorhagic stroke, kidney dysfunction, arthritis, or cognitive dysfunction, again suggesting bias towards minimizing harms.  

JUPITER(6)

Narrow, Unusual Patient Population - The study was limited to patients without hyperlipidemia but with an increased C-reactive protein.  Thus it is not clear that its results would generalize to a more typically defined primary prevention population.  The study excluded patients receiving post-menopausal hormone-replacement; with diabetes; uncontrolled hyertension; cancer other than non-melanoma skin cancer within 5 years; recent history of drug or alcohol abuse; "another medical condition that might compromise safety or the successful completion of the study;" also patients with "inflammatory conditions such as severe arthritis, lupus, or inflammatory bowel disease...;" and also "patients taking immunosuppressant agents such as cyclosporine, tacrolimus, azathioprine, or long-term oral glucocorticoids."  Again, this narrow patient population would reduce generalizability and bias towards minimizing harms.

Early Termination/ Multiple Comparisons - This study was terminated early after an early look at the data.  Allowing for multiple looks at the data may exaggerate efficacy.  

Implausible Dropout Rate - According to the Cochrane Review, the study had no dropouts.  This value seems implausible, again suggesting data collection problems.  

Adverse Effects Not Reported - The study provided no data about hemmorhagic stroke, arthritis, or cognitive dysfunction, again suggesting bias towards minimizing harms.  [Revised December 9, 2013 - see comment below by Marilyn Mann.]

Summary

Aspects of the continuing controversy over the new ACC/ AHA guidelines for statins in the primary prevention of cardiovascular disease hinted that the clinical trials which provided the evidentiary basis for the guidelines, and for the use of statins in primary prevention in general, was more flawed than is widely appreciated.  The latest Cochrane Collaboration review of this data acknowledged multiple, important flaws affecting most of the studies.  Our more detailed review of the three studies held out as the purest found additional flaws.  Many of these flaws seemed likely to bias the studies towards exaggerating the efficacy and/or minimizing the harms of statins in primary prevention.  Since all these trials were funded, and presumably influenced by pharmaceutical companies that make statins, these flaws seem to be examples of manipulation of the clinical evidence.  Rather than being the result of simple mistakes, or inevitable trade-offs, they seem to be study features intended the support the vested interests of the study sponsors.

It is not clear why the Cochrane review did not temper its conclusions based on the flaws in the studies, and particularly by the possibility that these flaws represented study manipulation.  

The multiple flaws, possibly due to study manipulation, in the clinical evidence about statins in primary prevention suggest that we should be extremely skeptical about whether the benefits of such treatment outweighs its harms, and hence about whether the recommendations in the latest guidelines to give statins to all patients predicted (but perhaps not accurately) to be at even slightly elevated risk are warranted.

The flaws in multiple large studies of a very common clinical problem, and their effects on systematic reviews and clinical practice guidelines suggest that suppression and manipulation of research are rife in medicine and health care, presumably fueled by the pervasive web of conflicts of interest that spans health care.  We need extreme skepticism about the integrity of clinical research, especially research sponsored by those whose products and services are being studied, and who thus have vested interests in the research turning out to make their products and services look good.  

The good news is that we may not have to look too far to find ways to improve the trustworthiness of guidelines and the soundness of medical decision making.  Implementation of the Institute of Medicine's recommendations on reducing conflicts of interest (look here), and developing trustworthy guidelines (look here) might lead to the development of sound guidelines in the future.  

Furthermore, while endless discoveries of manipulated and suppressed research may have lead some evidence-based medicine advocates to despair, our latest exercise suggests that the principles of evidence-based medicine, unflinchingly applied, could really do good.  Review of the three statin studies above based on standard principles of critical review readily spotted the multiple signs of manipulation.  The problem with the Cochrane review was not that it missed these signs.  Rather, the reviewers for some reason noted most of them, but then did not react.  If systematic reviews were done with sufficient skepticism about the possibility of manipulation of clinical research, and were willing to call out when the emperor seemed short on fabric, then a lot of mischief could be avoided.  


References

1.  Lenzer J. Majority of panelists on controversial new cholesterol guideline have current or recent ties to drug manufacturers.  Brit Med J 2013.  Link here. 

2.  Taylor FC, Huffman M, Shah E. Statin therapy for primary prevention of cardiovascular disease.  JAMA 2013.  Link here.  

3.  Robinson JG.  Accumulating evidence for statins in primary prevention.  JAMA 2013;  Link here.

4.    Downs  JR, Clearfield  M, Weis  S,  et al; for the AFCAPS/TexCAPS Research Group.  Primary prevention of acute coronary events with lovastatin in men and women with average cholesterol levels: results of AFCAPS/TexCAPS. JAMA. 1998;279(20):1615-1622. Link here. 
5.  Nakamura  H, Arakawa  K, Itakura  H,  et al; MEGA Study Group.  Primary prevention of cardiovascular disease with pravastatin in Japan (MEGA study): a prospective randomised controlled trial. Lancet. 2006;368(9542):1155-1163. Link here.

6.  Ridker  PM, Danielson  E, Fonseca  FA,  et al; JUPITER Study Group.  Rosuvastatin to prevent vascular events in men and women with elevated C-reactive protein. N Engl J Med. 2008;359(21):2195-2207. Link here. 

7.  Taylor F,  Huffman MD, Macedo AF et al.  Statins for the prevention of cardiovascular disease. Cochrane Library 2013.  Link here.

8. Mueller PS, Montori VM, Bassler D et al.  Ethical issues in stopping randomized trials early because of apparent benefit.  Ann Intern Med 2007; 146: 878-881.  Link here.  

9.  Management of Elevated Cholesterol in the Primary Prevention Group of the Adult Japanese (MEGA) Study Group.  Design and baseline characteristics of a study of primary prevention of coronary events with pravastatin among Japanese with mildly elevated cholesterol levels.  Circ J  2004; 68: 860-867.  Link here.