Why Bone-in-the-Nose Medicine Trumps Medical Science Most of the Time...,

medium  |  A fundamental tenet of science is that findings must be reproduced. One experiment does not establish new truths. The results have to be replicated by others using the methods described by the original investigators. Replication is key to ensuring that conclusions aren’t spurious. Nevertheless, science is currently plagued by hordes of irreproducible study results.

“ More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.”

Certainly, misidentification of cells is a major contributor to the replication crisis in basic biological science. However, statistics and publication bias combine to form another formidable pseudo-scientific edifice that churns out irreproducible results across scientific genres and misleads the public.

The infamous P-value lies at the heart of the matter. Simply put, the P-value is an arbitrary estimate of the likelihood that results of a given experiment are due to chance. The cutoff widely accepted across scientific disciplines is 5%. In other words, as long as the statistics say that the likelihood a given result is due to chance alone is 5% or less, then the result is considered “significant.” That might sound good at first glance, but when examined a little more closely, in conjunction with the concept of publication bias, the limitations rapidly mount.

The significance of the 5%, or .05, P-value is utterly arbitrary. A man named Ronald Fisher made it up back in the 1920s. It’s based on the rough approximation of how much of a normal (Gaussian) distribution will fall within two standard deviations of the mean — about 95%. (I’m not going to get into the problems with the normal distribution in this post, but I will recommend that anyone interested in this concept read Nassim Nicholas Taleb’s book The Black Swan.)

A P-value of .05 implies that one result in 20 will be due to chance. But how many millions of results are obtained from scientific experiments each year around the world? An incalculable number. It’s virtually guaranteed that thousands of results due to chance alone emerge from the realm of theory and intrude on what we presume to call reality each year. And those are the results that get published.

Scientists working in academia must, as the saying goes, publish or perish. And the journals in which those anxious scientists try to publish their results need to make money, which necessitates reader interaction. Results that are not “statistically significant” are boring. No reader wants to pay for a journal full of articles that say “we did this study using really careful methods, and nothing happened, it didn’t work. End of story.” If science were fully transparent, and results of all experiments were published, however, this is exactly what the vast majority of papers would say.

The failure of negative study results to ever see the light of day creates staggering wastes. It’s likely that many basic experiments have been repeated over and over again, with uninteresting results, and subsequently never published. Then, another research group comes along and does the experiment again (because they didn’t know about the previous null results) and, by chance alone, finds a positive result. Of course, that result is interesting and gets published. This basic cycle is why John Ioannidis’s now-famous 2005 paper was titled “Why Most Published Research Findings Are False.”