This essay, published in 2005 didn't have an effect on me at the time (I was just starting my MS degree at Pitt's GSPH). But once I found it during my PhD, it really resonated with me.
I was having problems in the lab with RT-qPCR and western blots, etc. and how to interpret results. One of the main issues was my advisor was so certain that his hypothesis from his grant was correct, he would often pre-write the manuscript. If my results didn't match his hypothesis, then clearly I was doing something technically incorrect. Since I was doing something technically incorrect, his answer was repetition. Over and over until the results matched his hypothesis.
When you run a RT-qPCR and it "fails" the first three times (or twenty), but succeeds the 4th, and you only use the data from the 4th, I think that those are disingenuous results, and I argue that they aren't valid.
Likewise when you run a western blot or other qualitative experiment, (IFA, IHC, etc.) if you have to run the experiment 10 times, and only see your desired results once or twice, just how valid is that experiment?
These were issues that I struggled with (and I assume many others do as well) so when I read Dr. Ioannidis' essay, I felt slightly validated in my opinion.
If you have questions, use the comments section below!
I want to discuss a few things about this essay. First, the summary:
"There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller, when effect sizes are smaller, when there is a greater number and lesser preselection of tested relationships, where there is greater flexibility in designs, definitions, outcomes, and analytic modes, when there is greater financial and other interest and prejudice, and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias."
Several methodologists have pointed out that the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0.05. Research is not most appropriately represented and summarized by p-values, but, unfortunately, there is a widespread notion that medical research articles should be interpreted based only on p-values.
- The smaller the studies conducted in a scientific field, the less likely the research findings are to be true. I agree!
- The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. I agree!
- The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true. I agree!
- The greater the flexibility in designs, definitions, outcomes, and analytic modes in a scientific field, the less likely the research findings are to be true. Eh, this one I could go either way on.
- The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true. I agree! However, if there are false results, there is a better than normal chance that another scientist overturns them.
- The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true. I disagree. I think this is the pinnacle of science. More teams working on the same findings. The author does call this a paradoxical corollary, but I don't know if I buy it. The more people validating research the better.
The author does have some suggestions about how to improve the situation.
- Better powered evidence, e.g. large studies or low-bias meta-analysis
- Diminishing bias through enhanced research standards and curtailing of prejudices
- Instead of chasing statistical significance, we should improve our understanding of the range of R-values, the pre-study odds
I would highly recommend anyone involved in research to read the paper itself. It is fairly accessible, but a working knowledge of statistics is helpful.
- Lance D. Presser has a PhD in microbiology and immunology and is a public health laboratorian.
- Hire Lance for any of your microbiology, virology, teaching, editing, grant writing, or public health consulting needs.
- Follow Lance @ldpsci