3 ways to spot a bad statistic | Mona Chalabi

Summary

This talk addresses the skepticism surrounding statistics, particularly those from government sources. The speaker emphasizes the importance of discerning reliable numbers and provides tools to evaluate them. They critique the overconfidence in polling data and suggest hand-drawn visualizations to communicate uncertainty. The talk also stresses the need to see oneself in the data and to question how it was collected, advocating for a nuanced understanding of statistics to inform public policy and debate.

Takeaways

• 🧐 **Skepticism in Numbers**: The speaker encourages skepticism towards numbers, especially those from government sources, to discern reliable from unreliable statistics.
• 📊 **Questioning Statistics**: The discussion highlights the importance of questioning statistics like unemployment rates, which are often distrusted by a significant portion of the population.
• 🌐 **Impact of Distrust**: Distrust in government economic data can lead to societal divisions and affect public policy, as people's relationships with these numbers can vary greatly.
• 📉 **The Need for Objectivity**: There's a call for the use of statistics to move beyond emotional anecdotes and measure societal progress in an objective manner.
• 🚫 **Critique of Elitism**: Some view government statistics as elitist or rigged, not reflecting everyday realities, leading to a debate on their validity and utility.
• 📈 **The Role of Data in Policymaking**: Good data is essential for creating fair policies; without it, it's challenging to address issues like discrimination, healthcare, and immigration.
• ❓ **Spotting Bad Statistics**: The speaker shares three key questions to ask when evaluating statistics: visibility of uncertainty, personal relevance in the data, and the method of data collection.
• 📋 **Uncertainty in Data**: The importance of recognizing and communicating uncertainty in data is emphasized, suggesting that data visualizations should reflect this to avoid misleading interpretations.
• 👁️ **Seeing Yourself in the Data**: Data should be relatable and relevant to individuals, prompting a deeper understanding of how it affects different segments of society.
• 🔍 **Methodology Matters**: The way data is collected is critical, and understanding methodologies can help identify potential biases or inaccuracies in the data.

Q & A

What is the speaker's overall goal in the talk?

The speaker's goal is to provide tools to help people distinguish between reliable and unreliable statistics. They want to encourage skepticism but also emphasize the importance of government data for informed decision-making.

Why does the speaker believe it's important to be skeptical of statistics?

The speaker argues that skepticism is crucial because numbers, especially those reported in the media or by private companies, can be misleading or manipulated. Being skeptical allows people to critically evaluate the accuracy and reliability of the statistics.

How does the speaker differentiate between private and government statistics?

The speaker explains that private companies may not prioritize accuracy and may present statistics in a way that benefits their agenda. In contrast, government statistics, while also subject to scrutiny, are generally collected impartially by civil servants and cover a much larger data set.

What is the problem with polling accuracy, according to the speaker?

The speaker points out several issues with polling accuracy, including the difficulty in obtaining a representative sample, people's reluctance to answer polls, and the possibility of respondents lying. Polling has become less reliable due to the diversity of modern societies.

Why does the speaker criticize how data is often visualized?

The speaker criticizes data visualizations for overstating certainty. They argue that sleek charts can make statistics seem more objective than they really are, which can numb people's critical thinking and make them accept the data without question.

What three questions does the speaker recommend asking to spot bad statistics?

The three questions are: 1) Can you see uncertainty in the data? 2) Can you see yourself in the data? 3) How was the data collected?

What does the speaker mean by asking 'Can you see uncertainty in the data?'

This question encourages people to assess whether the data accounts for variability or imprecision. The speaker argues that some visualizations make data appear more precise than it is, and it's important to recognize the uncertainty behind numbers.

What is the importance of asking 'Can I see myself in the data?'?

This question is about making sure that the data reflects personal or societal realities. It encourages people to question whether broad averages or generalizations match their own experiences or the experiences of others.

Why is understanding how data was collected crucial?

The methodology behind data collection can greatly impact the results. The speaker emphasizes that knowing how data was gathered—whether through a reliable process or not—helps determine the accuracy and reliability of the statistics.

What example does the speaker use to illustrate misleading data collection?

The speaker uses a poll that claimed 41% of Muslims in the US support jihad. Upon further examination, it turned out the majority defined 'jihad' as a personal struggle, not violent action. The poll also had methodological flaws, such as being an opt-in poll and having a very small sample size.

Outlines

00:00

📊 The Importance of Questioning Statistics

The speaker begins by acknowledging the skepticism some people have towards statistics, suggesting that it's healthy to question numbers, especially those from the government. They highlight the distrust in economic data, with a significant portion of Americans, and Trump supporters in particular, expressing doubt. The speaker emphasizes the need for discerning reliable numbers from unreliable ones and discusses the societal divides that stem from differing relationships with government statistics. They argue for the necessity of accurate statistics for policy-making and societal progress, while also advocating for critical engagement with these numbers.

05:00

📈 The Pitfalls of Data Visualization and Polling

In this section, the speaker critiques the use of political polls and their impact on public trust, arguing that they often overstate certainty and mislead the public. They discuss the inaccuracies in polling, such as the failure to capture diverse populations and the reluctance of people to participate honestly. The speaker also addresses the issue of data visualization, suggesting that charts can sometimes numb critical thinking. They propose hand-drawn visualizations as a more transparent method to communicate the imprecision of data, and they use examples like flu season probabilities and swimming pool fecal accidents to illustrate their point.

10:03

🗳️ Understanding Data Collection and Its Impact

The final paragraph delves into the importance of understanding how data is collected, arguing that it's as crucial as how it's communicated. The speaker uses an example of a poll suggesting a high percentage of Muslims supporting jihad, critiquing the methodology and highlighting how the definition of 'jihad' varied among respondents. They discuss the potential biases in opt-in polls and contrast them with government statistics, which are generally more reliable due to their larger sample sizes and impartiality. The speaker concludes by urging the audience not to dismiss numbers but to continually scrutinize them to ensure informed public policy decisions.

Keywords

💡Statistics

Statistics refers to the numerical data used to analyze, measure, and understand various aspects of society. In the video, the speaker emphasizes the importance of being skeptical yet informed about the statistics we encounter, especially those reported by the government or media.

💡Uncertainty

Uncertainty highlights the idea that not all numbers or statistics are precise. The speaker argues that showing uncertainty in data, such as through hand-drawn visualizations, helps people better understand the imprecision behind seemingly concrete numbers like political polls.

💡Polling

Polling is the process of gathering data to predict outcomes, often in politics. The speaker is critical of political polls, stating that they can be misleading due to sampling errors, societal diversity, and people’s reluctance to provide accurate responses.

💡Averages

Averages are a statistical measure often used to summarize data. The speaker explains that averages can be misleading, using examples like the average number of 'fecal accidents' in pools, which may not accurately represent the true distribution of incidents.

💡Data visualization

Data visualization refers to the graphical representation of data to make it easier to understand. The speaker warns that sleek, professional charts can create a false sense of certainty, while hand-drawn or imprecise visualizations may better convey the uncertainties in the data.

💡Government statistics

Government statistics are numerical data collected and published by government agencies. The speaker stresses their importance in forming fair public policies and criticizes moves to eliminate certain types of data collection, such as statistics on racial inequality.

💡Alternative facts

Alternative facts refer to distorted or misleading information that people use instead of objective data. The speaker highlights the current problem of 'alternative facts' dominating public discourse, causing people to mistrust or reject well-established statistics.

💡Representation in data

Representation in data means ensuring that the data reflects the experiences and perspectives of different groups in society. The speaker discusses how people can feel excluded from national statistics because averages do not always represent personal experiences, making it important to ask if one sees themselves in the data.

💡Methodology

Methodology refers to the process used to collect and analyze data. The speaker emphasizes that understanding how data is collected, such as whether a poll was opt-in or representative, is essential for assessing the validity of statistics.

💡Bias in statistics

Bias in statistics occurs when data is collected or presented in a way that favors certain outcomes or perspectives. The speaker illustrates this with an example of a poll about Muslims supporting jihad, where misreporting and selective presentation of results led to a distorted interpretation.

Highlights

The importance of being skeptical about statistics and learning to discern reliable numbers.

The difference between distrusting private company claims and questioning government statistics.

The high level of distrust in economic data among certain demographics, including Trump supporters.

The societal divide that starts to make sense when understanding people's relationships with government numbers.

The necessity of statistics for making sense of society and measuring progress objectively.

The argument that statistics are elitist and may not reflect everyday life experiences.

The current trend towards 'alternative facts' and the dismissal of statistics as a common ground for debate.

Legislative moves in the US to eliminate certain government statistics, such as those measuring racial inequality.

The critical role of data in observing and addressing discrimination, as well as in formulating fair policies.

The need to move beyond blindly accepting or rejecting government numbers and to learn how to spot bad statistics.

The challenges faced in a statistical department of the United Nations, particularly in data accuracy.

The idea that making numbers more accurate involves enabling more people to question them.

Three key questions to help spot bad statistics: seeing uncertainty, recognizing oneself in the data, and understanding data collection methods.

The impact of political polls on trust in numbers and media, and the inaccuracy of using polls to predict electoral outcomes.

The overstatement of certainty in data visualizations and the importance of communicating uncertainty.

The concept of averages and how they can be misleading, illustrated through a hand-drawn visualization of swimming pool fecal accidents.

The importance of seeing oneself in the data and understanding how statistics relate to personal experiences.

The significance of the axes in data visualization and how changing the scale can alter the story told by the data.

The importance of scrutinizing the data collection methods and the potential issues with opt-in polls.

The contrast between government and private statistics, with the former being more reliable due to larger sample sizes and impartiality.

The call to not give up on numbers but to question them critically to avoid making public policy decisions in the dark.

Transcripts

I'm going to be talking about statistics today.

If that makes you immediately feel a little bit wary, that's OK,

that doesn't make you some kind of crazy conspiracy theorist,

it makes you skeptical.

And when it comes to numbers, especially now, you should be skeptical.

But you should also be able to tell which numbers are reliable

and which ones aren't.

So today I want to try to give you some tools to be able to do that.

But before I do,

I just want to clarify which numbers I'm talking about here.

I'm not talking about claims like,

"9 out of 10 women recommend this anti-aging cream."

I think a lot of us always roll our eyes at numbers like that.

What's different now is people are questioning statistics like,

"The US unemployment rate is five percent."

What makes this claim different is it doesn't come from a private company,

it comes from the government.

About 4 out of 10 Americans distrust the economic data

that gets reported by government.

Among supporters of President Trump it's even higher;

it's about 7 out of 10.

I don't need to tell anyone here

that there are a lot of dividing lines in our society right now,

and a lot of them start to make sense,

once you understand people's relationships with these government numbers.

On the one hand, there are those who say these statistics are crucial,

that we need them to make sense of society as a whole

in order to move beyond emotional anecdotes

and measure progress in an [objective] way.

And then there are the others,

who say that these statistics are elitist,

maybe even rigged;

they don't make sense and they don't really reflect

what's happening in people's everyday lives.

It kind of feels like that second group is winning the argument right now.

We're living in a world of alternative facts,

where people don't find statistics this kind of common ground,

this starting point for debate.

This is a problem.

There are actually moves in the US right now

to get rid of some government statistics altogether.

Right now there's a bill in congress about measuring racial inequality.

The draft law says that government money should not be used

to collect data on racial segregation.

This is a total disaster.

If we don't have this data,

how can we observe discrimination,

let alone fix it?

In other words:

How can a government create fair policies

if they can't measure current levels of unfairness?

This isn't just about discrimination,

it's everything -- think about it.

How can we legislate on health care

if we don't have good data on health or poverty?

How can we have public debate about immigration

if we can't at least agree

on how many people are entering and leaving the country?

Statistics come from the state; that's where they got their name.

The point was to better measure the population

in order to better serve it.

So we need these government numbers,

but we also have to move beyond either blindly accepting

or blindly rejecting them.

We need to learn the skills to be able to spot bad statistics.

I started to learn some of these

when I was working in a statistical department

that's part of the United Nations.

Our job was to find out how many Iraqis had been forced from their homes

as a result of the war,

and what they needed.

It was really important work, but it was also incredibly difficult.

Every single day, we were making decisions

that affected the accuracy of our numbers --

decisions like which parts of the country we should go to,

who we should speak to,

which questions we should ask.

And I started to feel really disillusioned with our work,

because we thought we were doing a really good job,

but the one group of people who could really tell us were the Iraqis,

and they rarely got the chance to find our analysis, let alone question it.

So I started to feel really determined

that the one way to make numbers more accurate

is to have as many people as possible be able to question them.

So I became a data journalist.

My job is finding these data sets and sharing them with the public.

Anyone can do this, you don't have to be a geek or a nerd.

You can ignore those words; they're used by people

trying to say they're smart while pretending they're humble.

Absolutely anyone can do this.

I want to give you guys three questions

that will help you be able to spot some bad statistics.

So, question number one is: Can you see uncertainty?

One of things that's really changed people's relationship with numbers,

and even their trust in the media,

has been the use of political polls.

I personally have a lot of issues with political polls

because I think the role of journalists is actually to report the facts

and not attempt to predict them,

especially when those predictions can actually damage democracy

by signaling to people: don't bother to vote for that guy,

he doesn't have a chance.

Let's set that aside for now and talk about the accuracy of this endeavor.

Based on national elections in the UK, Italy, Israel

and of course, the most recent US presidential election,

using polls to predict electoral outcomes

is about as accurate as using the moon to predict hospital admissions.

No, seriously, I used actual data from an academic study to draw this.

There are a lot of reasons why polling has become so inaccurate.

Our societies have become really diverse,

which makes it difficult for pollsters to get a really nice representative sample

of the population for their polls.

People are really reluctant to answer their phones to pollsters,

and also, shockingly enough, people might lie.

But you wouldn't necessarily know that to look at the media.

For one thing, the probability of a Hillary Clinton win

was communicated with decimal places.

We don't use decimal places to describe the temperature.

How on earth can predicting the behavior of 230 million voters in this country

be that precise?

And then there were those sleek charts.

See, a lot of data visualizations will overstate certainty, and it works --

these charts can numb our brains to criticism.

When you hear a statistic, you might feel skeptical.

As soon as it's buried in a chart,

it feels like some kind of objective science,

and it's not.

So I was trying to find ways to better communicate this to people,

to show people the uncertainty in our numbers.

What I did was I started taking real data sets,

and turning them into hand-drawn visualizations,

so that people can see how imprecise the data is;

so people can see that a human did this,

a human found the data and visualized it.

For example, instead of finding out the probability

of getting the flu in any given month,

you can see the rough distribution of flu season.

This is --

(Laughter)

a bad shot to show in February.

But it's also more responsible data visualization,

because if you were to show the exact probabilities,

maybe that would encourage people to get their flu jabs

at the wrong time.

The point of these shaky lines

is so that people remember these imprecisions,

but also so they don't necessarily walk away with a specific number,

but they can remember important facts.

Facts like injustice and inequality leave a huge mark on our lives.

Facts like Black Americans and Native Americans have shorter life expectancies

than those of other races,

and that isn't changing anytime soon.

Facts like prisoners in the US can be kept in solitary confinement cells

that are smaller than the size of an average parking space.

The point of these visualizations is also to remind people

of some really important statistical concepts,

concepts like averages.

So let's say you hear a claim like,

"The average swimming pool in the US contains 6.23 fecal accidents."

That doesn't mean every single swimming pool in the country

contains exactly 6.23 turds.

So in order to show that,

I went back to the original data, which comes from the CDC,

who surveyed 47 swimming facilities.

And I just spent one evening redistributing poop.

So you can kind of see how misleading averages can be.

(Laughter)

OK, so the second question that you guys should be asking yourselves

to spot bad numbers is:

Can I see myself in the data?

This question is also about averages in a way,

because part of the reason why people are so frustrated

with these national statistics,

is they don't really tell the story of who's winning and who's losing

from national policy.

It's easy to understand why people are frustrated with global averages

when they don't match up with their personal experiences.

I wanted to show people the way data relates to their everyday lives.

I started this advice column called "Dear Mona,"

where people would write to me with questions and concerns

and I'd try to answer them with data.

People asked me anything.

questions like, "Is it normal to sleep in a separate bed to my wife?"

"Do people regret their tattoos?"

"What does it mean to die of natural causes?"

All of these questions are great, because they make you think

about ways to find and communicate these numbers.

If someone asks you, "How much pee is a lot of pee?"

which is a question that I got asked,

you really want to make sure that the visualization makes sense

to as many people as possible.

These numbers aren't unavailable.

Sometimes they're just buried in the appendix of an academic study.

And they're certainly not inscrutable;

if you really wanted to test these numbers on urination volume,

you could grab a bottle and try it for yourself.

(Laughter)

The point of this isn't necessarily

that every single data set has to relate specifically to you.

I'm interested in how many women were issued fines in France

for wearing the face veil, or the niqab,

even if I don't live in France or wear the face veil.

The point of asking where you fit in is to get as much context as possible.

So it's about zooming out from one data point,

like the unemployment rate is five percent,

and seeing how it changes over time,

or seeing how it changes by educational status --

this is why your parents always wanted you to go to college --

or seeing how it varies by gender.

Nowadays, male unemployment rate is higher

than the female unemployment rate.

Up until the early '80s, it was the other way around.

This is a story of one of the biggest changes

that's happened in American society,

and it's all there in that chart, once you look beyond the averages.

The axes are everything;

once you change the scale, you can change the story.

OK, so the third and final question that I want you guys to think about

when you're looking at statistics is:

How was the data collected?

So far, I've only talked about the way data is communicated,

but the way it's collected matters just as much.

I know this is tough,

because methodologies can be opaque and actually kind of boring,

but there are some simple steps you can take to check this.

I'll use one last example here.

One poll found that 41 percent of Muslims in this country support jihad,

which is obviously pretty scary,

and it was reported everywhere in 2015.

When I want to check a number like that,

I'll start off by finding the original questionnaire.

It turns out that journalists who reported on that statistic

ignored a question lower down on the survey

that asked respondents how they defined "jihad."

And most of them defined it as,

"Muslims' personal, peaceful struggle to be more religious."

Only 16 percent defined it as, "violent holy war against unbelievers."

This is the really important point:

based on those numbers, it's totally possible

that no one in the survey who defined it as violent holy war

also said they support it.

Those two groups might not overlap at all.

It's also worth asking how the survey was carried out.

This was something called an opt-in poll,

which means anyone could have found it on the internet and completed it.

There's no way of knowing if those people even identified as Muslim.

And finally, there were 600 respondents in that poll.

There are roughly three million Muslims in this country,

according to Pew Research Center.

That means the poll spoke to roughly one in every 5,000 Muslims

in this country.

This is one of the reasons

why government statistics are often better than private statistics.

A poll might speak to a couple hundred people, maybe a thousand,

or if you're L'Oreal, trying to sell skin care products in 2005,

then you spoke to 48 women to claim that they work.

(Laughter)

Private companies don't have a huge interest in getting the numbers right,

they just need the right numbers.

Government statisticians aren't like that.

In theory, at least, they're totally impartial,

not least because most of them do their jobs regardless of who's in power.

They're civil servants.

And to do their jobs properly,

they don't just speak to a couple hundred people.

Those unemployment numbers I keep on referencing

come from the Bureau of Labor Statistics,

and to make their estimates,

they speak to over 140,000 businesses in this country.

I get it, it's frustrating.

If you want to test a statistic that comes from a private company,

you can buy the face cream for you and a bunch of friends, test it out,

if it doesn't work, you can say the numbers were wrong.

But how do you question government statistics?

You just keep checking everything.

Find out how they collected the numbers.

Find out if you're seeing everything on the chart you need to see.

But don't give up on the numbers altogether, because if you do,

we'll be making public policy decisions in the dark,

using nothing but private interests to guide us.

Thank you.

(Applause)