3 ways to spot a bad statistic | Mona Chalabi

Summary

This talk addresses the skepticism surrounding statistics, particularly those from government sources. The speaker emphasizes the importance of discerning reliable numbers and provides tools to evaluate them. They critique the overconfidence in polling data and suggest hand-drawn visualizations to communicate uncertainty. The talk also stresses the need to see oneself in the data and to question how it was collected, advocating for a nuanced understanding of statistics to inform public policy and debate.

Takeaways

• 🧐 **Skepticism in Numbers**: The speaker encourages skepticism towards numbers, especially those from government sources, to discern reliable from unreliable statistics.
• 📊 **Questioning Statistics**: The discussion highlights the importance of questioning statistics like unemployment rates, which are often distrusted by a significant portion of the population.
• 🌐 **Impact of Distrust**: Distrust in government economic data can lead to societal divisions and affect public policy, as people's relationships with these numbers can vary greatly.
• 📉 **The Need for Objectivity**: There's a call for the use of statistics to move beyond emotional anecdotes and measure societal progress in an objective manner.
• 🚫 **Critique of Elitism**: Some view government statistics as elitist or rigged, not reflecting everyday realities, leading to a debate on their validity and utility.
• 📈 **The Role of Data in Policymaking**: Good data is essential for creating fair policies; without it, it's challenging to address issues like discrimination, healthcare, and immigration.
• ❓ **Spotting Bad Statistics**: The speaker shares three key questions to ask when evaluating statistics: visibility of uncertainty, personal relevance in the data, and the method of data collection.
• 📋 **Uncertainty in Data**: The importance of recognizing and communicating uncertainty in data is emphasized, suggesting that data visualizations should reflect this to avoid misleading interpretations.
• 👁️ **Seeing Yourself in the Data**: Data should be relatable and relevant to individuals, prompting a deeper understanding of how it affects different segments of society.
• 🔍 **Methodology Matters**: The way data is collected is critical, and understanding methodologies can help identify potential biases or inaccuracies in the data.

Q & A

What is the speaker's overall goal in the talk?

The speaker's goal is to provide tools to help people distinguish between reliable and unreliable statistics. They want to encourage skepticism but also emphasize the importance of government data for informed decision-making.

Why does the speaker believe it's important to be skeptical of statistics?

The speaker argues that skepticism is crucial because numbers, especially those reported in the media or by private companies, can be misleading or manipulated. Being skeptical allows people to critically evaluate the accuracy and reliability of the statistics.

How does the speaker differentiate between private and government statistics?

The speaker explains that private companies may not prioritize accuracy and may present statistics in a way that benefits their agenda. In contrast, government statistics, while also subject to scrutiny, are generally collected impartially by civil servants and cover a much larger data set.

What is the problem with polling accuracy, according to the speaker?

The speaker points out several issues with polling accuracy, including the difficulty in obtaining a representative sample, people's reluctance to answer polls, and the possibility of respondents lying. Polling has become less reliable due to the diversity of modern societies.

Why does the speaker criticize how data is often visualized?

The speaker criticizes data visualizations for overstating certainty. They argue that sleek charts can make statistics seem more objective than they really are, which can numb people's critical thinking and make them accept the data without question.

What three questions does the speaker recommend asking to spot bad statistics?

The three questions are: 1) Can you see uncertainty in the data? 2) Can you see yourself in the data? 3) How was the data collected?

What does the speaker mean by asking 'Can you see uncertainty in the data?'

This question encourages people to assess whether the data accounts for variability or imprecision. The speaker argues that some visualizations make data appear more precise than it is, and it's important to recognize the uncertainty behind numbers.

What is the importance of asking 'Can I see myself in the data?'?

This question is about making sure that the data reflects personal or societal realities. It encourages people to question whether broad averages or generalizations match their own experiences or the experiences of others.

Why is understanding how data was collected crucial?

The methodology behind data collection can greatly impact the results. The speaker emphasizes that knowing how data was gathered—whether through a reliable process or not—helps determine the accuracy and reliability of the statistics.

What example does the speaker use to illustrate misleading data collection?

The speaker uses a poll that claimed 41% of Muslims in the US support jihad. Upon further examination, it turned out the majority defined 'jihad' as a personal struggle, not violent action. The poll also had methodological flaws, such as being an opt-in poll and having a very small sample size.