What do my results mean Effect size is not the same as statistical significance. With Tom Reader.

Summary

In this video, Tom Reader explores the difference between effect size and statistical significance, using two hypothetical hair growth treatments as examples. He critiques the claims of Furball Express, which lacks robust evidence, and contrasts it with Toupee Away, which shows statistical significance but minimal practical effect. Reader emphasizes that a p-value alone is insufficient for interpreting study results; understanding the actual size and relevance of the effect is crucial. Ultimately, he advises researchers to provide comprehensive data, ensuring clarity about their findings' implications.

Takeaways

• 😀 Effect size and statistical significance are crucial concepts in interpreting research results.
• 😀 The p-value indicates whether to reject the null hypothesis but doesn't convey the effect's size or importance.
• 😀 Example comparisons of two hair growth treatments highlight the difference between significant results and meaningful outcomes.
• 😀 A small sample size can lead to misleading conclusions, as seen with Furball Express's poor experimental design.
• 😀 Larger studies with proper randomization yield more reliable data, as shown with the Toupee Away trial.
• 😀 Even statistically significant results can indicate trivial effects, exemplified by the minimal hair growth from Toupee Away.
• 😀 Reporting only p-values can be misleading; researchers must also provide effect sizes for clarity.
• 😀 The importance of an effect can vary greatly; a small percentage reduction in mortality can still have major implications.
• 😀 Researchers should express results in ways that reflect their significance and relevance, such as percentages or R-squared values.
• 😀 Clear communication of research findings is essential for proper public understanding and application.

Q & A

What is the main difference between effect size and statistical significance?

Effect size measures the magnitude of a difference or relationship, while statistical significance indicates whether an observed effect is likely due to chance.

Why is p-value important in statistical analysis?

The p-value helps determine whether to reject the null hypothesis by indicating the probability that the observed results occurred by chance.

What issue is highlighted regarding the sample size in the Furball Express study?

The Furball Express study had a very small sample size of only eight volunteers, which limits the reliability and generalizability of its findings.

What does a p-value of less than 0.01 indicate in the Toupee Away study?

A p-value of less than 0.01 indicates that there is less than a 1% chance that the observed difference in hair growth occurred by chance, suggesting a statistically significant result.

Why might a statistically significant result still be practically irrelevant?

A result can be statistically significant but show a very small effect size, making it practically irrelevant, as seen with the minimal hair growth from Toupee Away.

What kind of data was used to evaluate the effectiveness of Toupee Away?

Toupee Away's effectiveness was evaluated using precise measurements of hair length taken from a large sample of 1000 volunteers before and after treatment.

How should researchers report their findings for clarity?

Researchers should report both the p-value and the effect size, ensuring that audiences understand both the significance and the practical implications of their results.

What measurement tool was criticized in the Furball Express study?

The use of a ruler to measure hair growth was criticized for its lack of precision, particularly since hair growth can vary greatly based on where measurements are taken.

What is an example of a meaningful small effect in scientific studies?

A 2% reduction in deaths from malaria can be a small effect statistically, but it has significant real-world implications, potentially saving thousands of lives.

What alternative methods can researchers use to express effect size?

Researchers can express effect size as a percentage difference between means or use r-squared values to indicate how much variance in one variable is explained by another.