Is Most Published Research Wrong?

Summary

This script from 'Journal of Personality and Social Psychology' discusses the concept of precognition and the reliability of scientific research. It critiques the use of p-values, highlighting how they can lead to false positives and the reproducibility crisis in science. The script also touches on the incentives for scientists to publish significant findings, the challenges of replication, and recent efforts to improve research integrity, emphasizing the importance of skepticism and rigorous methodology in scientific inquiry.

Takeaways

• 🔮 In 2011, a study titled 'Feeling the Future' suggested that people might have the ability to see into the future, but the results were not conclusive.
• 🎰 The study involved predicting images behind curtains, with a hit rate of 53% for erotic images, which was statistically significant with a p-value of .01.
• 📊 Scientists use p-values to determine the significance of results, with a threshold of .05 commonly used to indicate that results are unlikely due to chance.
• 🧐 However, relying on a p-value of .05 can lead to a significant number of false positives, especially when multiple hypotheses are being tested.
• 🧪 The 'Reproducibility Project' found that only 36% of psychology studies could be statistically significantly replicated, raising questions about the reliability of published research.
• 🍫 A study supposedly showing that eating chocolate helps with weight loss was later revealed to be a case of p-hacking, where data was manipulated to produce a false positive result.
• 🔬 The scientific community has recognized the reproducibility crisis and is taking steps to improve research practices, such as pre-registering studies and reducing publication bias.
• 📈 Publication bias is a significant issue, as journals are more likely to publish studies with statistically significant results, which can skew the scientific record.
• 📉 The pressure to publish can lead researchers to focus on novel and unexpected hypotheses, which may not hold up under scrutiny, further contributing to the reproducibility crisis.
• 🌟 Despite the challenges, science remains the most reliable method for understanding the world, even if it is not infallible.

Q & A

What was the title of the article published in the Journal of Personality and Social Psychology in 2011?

The title of the article was 'Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect'.

What was the main claim of the 2011 study regarding the ability to see into the future?

The main claim of the study was that people could potentially see into the future, as indicated by participants having a slightly higher hit rate than chance when predicting which curtain hid an erotic image.

What is a p-value in the context of statistical significance?

A p-value is a statistical measure that indicates the probability of obtaining a result at least as extreme as the one observed, assuming the null hypothesis is true. A p-value less than .05 is generally considered significant.

Why might a 53% hit rate for erotic images in the study not necessarily mean that people can see into the future?

A 53% hit rate does not necessarily mean that people can see into the future because it could be due to chance, and the p-value of .01, while significant, might not be enough to conclude such an extraordinary claim without further evidence.

What is the issue with using a p-value threshold of .05 for determining statistical significance?

Using a p-value threshold of .05 can lead to a significant number of false positives, especially when there are many hypotheses being tested or when the studies are underpowered, biased, or suffer from p-hacking.

What is meant by 'p-hacking' in the context of scientific research?

P-hacking refers to the practice of manipulating the data analysis, such as selecting or modifying variables, sample sizes, or statistical techniques, to achieve a p-value that meets the threshold for statistical significance.

What was the result of the Reproducibility Project that attempted to replicate 100 psychology studies?

The Reproducibility Project found that only 36% of the 100 psychology studies they attempted to replicate had statistically significant results the second time around.

Why are replication studies important in scientific research?

Replication studies are important because they help to verify the validity and reliability of initial findings, ensuring that scientific knowledge is built on solid and reproducible evidence.

What are some of the recent changes in scientific practices aimed at addressing the reproducibility crisis?

Recent changes include conducting large-scale replication studies, establishing sites like Retraction Watch for withdrawn papers, creating repositories for unpublished negative results, and adopting practices like preregistration of hypotheses and methods to reduce publication bias and p-hacking.

What is the significance of the pentaquark example in the context of the reproducibility crisis?

The pentaquark example illustrates how even with stringent statistical requirements, false discoveries can occur due to biases in data analysis, emphasizing the importance of blinding and replication in scientific research.