News outlets love “breakthrough” stories about medical research studies, which seem to promise a new cure almost every week. A new report shows how the scientists who conduct such research sometimes add to the hype. The problem, according to a researcher at the Stanford Prevention Research Center, is that research claiming sweeping effect usually is wrong. Or at least premature.
As published in the Journal of the American Medical Association (JAMA), the study by John Ioannidis and his team analyzed data from thousands of medical trials and concluded that most interventions have a small or modest incremental effect and that such effects often are overestimated by small studies.
“Typically,” the researchers wrote, “trials with very large effects have limited evidence. … When additional evidence is obtained, most of the very large treatment effects become much smaller and many lose their nominal significance.”
This inflation of study results doesn’t even address another problem with research made public-the willful misinterpretation of otherwise objective data. (See our post about manipulating study results.)
Scientists conduct clinical trials to test the effectiveness, side effects and potential harm of a new medicine, device or treatment regimen. The most rigorous tests involve subjects who are randomized (with a coin flip or some other random choice) between the treatment being studied and a “control” treatment, which is often a placebo. When possible, the trial is conducted so that neither the subjects nor the researchers know which group gets the treatment and which is the control group. (In the case of a new drug, the latter group is given a placebo, or inert substance, instead of the medicine, or some established medicine with known results; for a device study, the control group gets the current best treatment.)
Ioannidis’ Stanford research center works to improve the testing process and draw attention to its shortcomings. His mission in the JAMA study was to generate data indicating how clinical trials that report major effects sustain their success over time. The study, which reviewed articles on health-care studies around the world, showed that, generally, they don’t.
Their definition of “major effect” was a positive or negative impact that was five times greater than what the control group experienced. Approximately 1 in 10 of the medical topics they studied concluded that the treatment effect was major. But only about 6 in 100 repeated the same, significant effect in a later trial. And in more than 9 in 10 of all those cases, the large effect vanished with additional studies or more detailed analysis of the data.
Anyone who completed middle-school science knows that you don’t get to claim your research is a breakthrough if you can’t replicate the results in a later study.
Only one out of the thousands of interventions the JAMA researchers studied that involved randomized trials had a major effect on mortality, significant statistical heft and no concerns about its validity. It involved a treatment for respiratory failure in newborns. Others that were both statistically significant and valid were related not to life-threatening conditions, but to minor symptoms or involved harms from the treatment. And there were only about 9 in 100 of those.
Almost all of the trials that claimed very large effects were small, usually involving fewer than 20 subjects. There’s no way to know with such a small group of patients whether the results are real or the result of chance. So larger studies are required to prove that the major effects from the smaller ones are legitimate and meaningful.
The message is that any consumer of a media report that some small, single study made a huge medical breakthrough should be immediately skeptical. We’ve long railed against the hype that proliferates in medical news for consumers, whether it concerns medical screening, drugs or supplements or disease-mongering.
Now, as this study shows, overhyping the results of a study can be as much the fault of the scientist who selectively uses data as it is the media that savors a sexy story.
Medicine is ripe for over-exuberance: Remember, if it sounds too good to be true, it probably is.