p-values: What they are and how to interpret them

Summary

In this StatQuest episode, Josh Starmer explains the concept of p-values in the context of comparing two drugs. He illustrates how a small sample size can lead to unreliable conclusions, emphasizing the importance of testing on a larger group to increase confidence in results. The video clarifies that a low p-value (commonly below 0.05) indicates strong evidence against the null hypothesis, suggesting a significant difference between the drugs. However, p-values do not measure the magnitude of the difference, and a small p-value doesn't guarantee a large effect size.

Takeaways

• 🔍 P-values help quantify the confidence in the difference between two groups.
• 💊 Comparing two drugs requires testing on many people to avoid random errors.
• 📊 Larger sample sizes provide more reliable results.
• 🎯 A p-value closer to zero indicates higher confidence in the difference between groups.
• 🛑 The common threshold for p-values is 0.05, meaning a 5% chance of a false positive.
• 🧪 Random events can affect results, which is why larger sample sizes are crucial.
• 🧮 P-values don't measure the size of the difference, just the confidence in the difference.
• 📈 Hypothesis testing uses p-values to reject or accept the null hypothesis.
• ⚠️ Smaller thresholds reduce false positives but may not always be practical.
• 🔬 The p-value is affected by both the sample size and the magnitude of the effect.

Q & A

What is the main topic of the video script?

The main topic of the video script is explaining what p-values are and how to interpret them in the context of statistical analysis, particularly in comparing the effectiveness of two drugs.

Why can't we conclude that Drug A is better than Drug B after testing on just one person each?

We can't conclude that Drug A is better than Drug B after testing on just one person each because there could be many confounding factors such as medication interactions, allergies, improper dosage, or placebo effects that could affect the outcome.

What is the purpose of testing drugs on more than one person?

The purpose of testing drugs on more than one person is to account for the variability and randomness in individual responses, which helps to reduce the impact of confounding factors and provides a more reliable basis for comparison.

What does a p-value represent in statistical testing?

A p-value represents the probability that the observed results occurred by chance if there is actually no difference between the groups being compared, such as two different drugs.

Why is a commonly used threshold for p-values set at 0.05?

A commonly used threshold of 0.05 for p-values means that if there is no actual difference between the groups, only 5% of the experiments would incorrectly conclude a difference, thus balancing the trade-off between false positives and the cost of further testing.

What is a false positive in the context of p-values?

A false positive occurs when a small p-value is obtained even though there is no actual difference between the groups, indicating a statistically significant result that is not supported by a true difference.

What does a p-value not tell us about the drugs being tested?

A p-value does not tell us about the effect size or the actual difference between the drugs. It only indicates the probability that the observed difference is due to chance.

What is the null hypothesis in hypothesis testing?

The null hypothesis is the starting assumption in hypothesis testing, which states that there is no difference between the groups being compared, such as the effectiveness of two drugs.

Why might a study with a large effect size have a larger p-value than expected?

A study with a large effect size might have a larger p-value than expected if the sample size is small, which reduces the statistical power to detect a significant difference.

How can changing the threshold for p-values affect the likelihood of a false positive?

Changing the threshold for p-values to a smaller number reduces the likelihood of a false positive by requiring stronger evidence against the null hypothesis, while a larger threshold increases the likelihood of a false positive.