Why The Polls Keep Getting It Wrong - Cheddar Explains

Summary

This video explores the evolution and challenges of political polling, from informal newspaper surveys to George Gallup's revolutionary 1936 methods. Using quota sampling and the famous 'beans in a barrel' analogy, Gallup accurately predicted Franklin Roosevelt's victory, highlighting the importance of representative samples. Modern polling faces new obstacles: declining landline use, expensive cell phone outreach, low response rates, and weighting complexities. While polls effectively measure preferences, predicting voter behavior remains difficult. The video underscores that despite technological and societal changes, the principles of accurate sampling are as crucial today as they were nearly a century ago, shaping how we understand public opinion.

Takeaways

• 🗳️ Polling accuracy depends heavily on having a representative sample, not just a large number of respondents.
• 📜 Early polls, like those by the Literary Digest, failed in 1936 because their sample overrepresented older, wealthier, and more conservative voters.
• 👨‍🔬 George Gallup introduced quota sampling, ensuring survey samples mirrored the population’s demographics, which improved accuracy.
• 🍲 Sampling is like tasting soup: a small, well-chosen sample can reflect the whole if it is properly mixed.
• ⚖️ Modern challenges, such as the shift from landlines to cell phones, make it harder to reach a representative sample.
• 📵 Laws against auto-dialing cell phones increase the cost and complexity of conducting surveys today.
• 📉 Response rates have plummeted from around 80% in the 1970s to roughly 8% today, which undermines poll reliability.
• 🔧 Weighting is used to adjust for underrepresented groups, but if respondents are outliers, this can skew results.
• 📊 Polls are better at measuring public opinion than predicting actual voter behavior, which remains difficult.
• ⏳ Despite challenges, polling remains a valuable tool for tracking trends, even if precise election predictions can fail.
• 🧾 Historical polling mistakes, such as the 1948 Dewey-Truman prediction, highlight the difficulty of forecasting elections accurately.
• 👥 Including diverse demographic groups—across age, occupation, and ethnicity—is crucial for accurate survey results.

Q & A

What was the main issue with the Literary Digest's 1936 poll predicting the US presidential election?

The Literary Digest's poll was biased because it drew its sample from phone books and vehicle registration lists, which skewed older, wealthier, and more conservative, underrepresenting Franklin Roosevelt supporters.

How did George Gallup's method differ from the Literary Digest's approach?

Gallup used quota sampling, ensuring his sample reflected the population's demographics proportionally, rather than relying on large but biased mailing lists.

What analogy did Gallup use to explain sampling public opinion?

Gallup compared sampling public opinion to sampling soup: a properly selected spoonful can reflect the taste of the whole pot, meaning you don’t need to survey everyone, just the right subgroups.

Why is the sample considered more important than the sample size in polling?

Because a representative sample captures the diversity of the population accurately, while a large but biased sample can give misleading results, as seen in the Literary Digest case.

What modern challenge affects reaching poll respondents using phones?

Most people now primarily use cell phones, and automated dialing of cell phones is restricted by law, requiring expensive manual dialing and making it harder to reach certain demographics.

How have response rates changed over time and why does it matter?

Response rates have dropped from about 80% in the 1970s to around 8% today. Low response rates make polls less accurate because fewer responses increase the risk of unrepresentative samples.

What is 'weighting' in polling and what risk does it carry?

Weighting adjusts responses to better reflect the population when some groups are underrepresented. The risk is that a single respondent may be weighted heavily, potentially skewing results if they are an outlier.

Why is predicting voter behavior harder than measuring voter preference?

People can state which candidate they prefer, but predicting whether they will actually vote requires forecasting human behavior, which is much more variable and less predictable.

What historical example shows that even Gallup's method could fail?

In the 1948 presidential election, polls (including Gallup's) incorrectly predicted Thomas Dewey would win over Harry Truman, highlighting the inherent difficulty in predicting outcomes despite accurate sampling methods.

Why does the transcript suggest polling remains valuable despite its challenges?

Polls provide insights into public opinion trends and preferences, which are useful for understanding societal attitudes, even if they cannot perfectly predict election results or human behavior.