That’s the provocative headline on an article by an internationally regarded skeptic of medical research. And the striking thing is that many researchers agree their field is badly flawed.
Dr. John Ioannidis bolstered his contention about the wrongness of most published research with an elaborate mathematical proof published in the on-line journal PLoS Medicine.
Anyone who follows the medical headlines even casually and has a decent memory knows that Dr. Ioannidis is right. Whether the issue is cancer screening with PSA or mammograms, or nutritional research on the value of fish oil, or a hundred other subjects, the arc of medical knowledge follows a predictable path: from excitement to widespread adoption to more careful research to disillusionment.
Here’s Dr. Ioannidis’s own summary of the factors that go into the wrongness of most published research, in his PLoS Medicine essay:
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
A more accessible discussion of Dr. Ioannidis’s work is published in The Atlantic this month, by David Freedman. An excerpt:
Studies have gone back and forth on the cancer-preventing powers of vitamins A, D, and E; on the heart-health benefits of eating fat and carbs; and even on the question of whether being overweight is more likely to extend or shorten your life. How should we choose among these dueling, high-profile nutritional findings? Ioannidis suggests a simple approach: ignore them all.
For starters, he explains, the odds are that in any large database of many nutritional and health factors, there will be a few apparent connections that are in fact merely flukes, not real health effects-it’s a bit like combing through long, random strings of letters and claiming there’s an important message in any words that happen to turn up. …
Even if changing that one factor does bring on the claimed improvement, there’s still a good chance that it won’t do you much good in the long run, because these studies rarely go on long enough to track the decades-long course of disease and ultimately death. Instead, they track easily measurable health “markers” such as cholesterol levels, blood pressure, and blood-sugar levels, and meta-experts have shown that changes in these markers often don’t correlate as well with long-term health as we have been led to believe.
On the relatively rare occasions when a study does go on long enough to track mortality, the findings frequently upend those of the shorter studies.
And so it goes for all medical studies, he says. Indeed, nutritional studies aren’t the worst. Drug studies have the added corruptive force of financial conflict of interest.