TEDxMidAtlantic 2011 - Duncan Watts - The Myth of Common Sense

Summary

The speaker, a former physicist turned sociologist, discusses the challenges of solving complex social problems compared to fields like physics. While common sense helps in everyday situations, it often fails with large-scale social issues involving many individuals interacting in complex ways. The speaker advocates for a more scientific approach to social science, leveraging data from online interactions to better understand human behavior. They highlight the transformative potential of data-driven methods, despite the inherent messiness of social systems, and emphasize the need for systematic research to tackle social problems more effectively.

Takeaways

• 🧑‍🔬 The speaker has a multidisciplinary background, starting in physics and moving into sociology and social networks.
• 🌍 Social problems are more complex than 'rocket science' because they involve many interacting individuals with unpredictable behaviors.
• 💡 Common sense helps us navigate everyday situations but can mislead us when dealing with complex, large-scale problems like economics or politics.
• 🔄 Humans often view complex social issues with a bias of obviousness—once we know the outcome, we can easily craft explanations that seem inevitable.
• 🤔 The real issue with common sense is that when every possible explanation seems obvious, we lose the ability to accurately analyze complex situations.
• 📚 Historical narratives often mislead us, as we try to generalize from past events to predict future outcomes, even though history rarely repeats itself in the same way.
• 🔬 The scientific method has been essential in solving problems where intuition falls short, especially in areas where human behavior is involved.
• 📊 The rise of the internet and digital data has allowed sociologists to measure and analyze human interactions at unprecedented scales.
• 🔭 The availability of large-scale social data is like a 'telescope' for social science, making invisible patterns of behavior visible for analysis.
• 🚀 While social science may never have the elegant laws of physics, the speaker believes that data-driven approaches can significantly improve our understanding of complex social issues.

Q & A

What is the speaker's academic background?

The speaker started as a physicist in Australia, writing an undergraduate thesis on Chaos Theory. Later, they moved to the U.S., completed a PhD in engineering at Cornell University, and transitioned to sociology, eventually working at Yahoo Research.

How does the speaker describe the differences between social science and other scientific disciplines like physics?

The speaker notes that while problems in physics or chemistry often seem difficult, problems in sociology seem easier at first glance, but this perception is misleading. Social problems are complex, involve human behavior, and are challenging to solve, unlike the more deterministic nature of rocket science.

What example does the speaker give to illustrate the difficulty in solving social problems?

The speaker mentions examples like the ongoing difficulty in measuring the effectiveness of advertising, solving political crises, predicting financial meltdowns, and effectively aiding developing nations, despite centuries of study and effort.

What is the central claim the speaker makes about common sense?

The speaker argues that common sense works well for everyday, concrete situations but can mislead us when applied to complex, large-scale social problems involving many interacting people over time.

How does the speaker explain the failure of common sense in understanding complex social problems?

Common sense helps in routine decision-making and interpersonal interactions, but it struggles with large-scale issues that involve complex, dynamic systems. This reliance on common sense leads to incorrect assumptions and conclusions about social problems.

What historical example does the speaker use to demonstrate the flaws of common sense?

The speaker references a 1947 review by sociologist Paul Lazarsfeld about a study on American soldiers during World War II. Lazarsfeld shows how, regardless of the study’s results, people can justify either outcome as 'obvious,' which highlights the problem with relying on obviousness or common sense.

Why does the speaker criticize the way people interpret historical events?

The speaker argues that hindsight bias causes people to construct narratives that make the outcome of events seem inevitable. These narratives, however, are just stories, not true explanations of why events occurred.

What does the speaker say about the limitations of learning from history?

While people often say that those who do not learn from history are doomed to repeat it, the speaker points out that history never truly repeats itself in complex systems, making it difficult to use past events to predict future outcomes accurately.

What potential solution does the speaker offer to improve our understanding of social problems?

The speaker advocates for a more scientific, systematic approach to studying social problems, leveraging data and computational methods. This approach can offer insights where common sense fails, especially as modern technology allows for the measurement of human interactions on a large scale.

How has technology changed social science, according to the speaker?

The speaker highlights that the internet and digital platforms now allow social scientists to observe and measure large-scale human interactions in real time, which was previously impossible. This data-driven approach is transforming social science.

What does the speaker predict for the future of social science?

While social science is unlikely to achieve the same level of predictive power as physics, due to the complexity of human behavior, the speaker believes the rise of data and computational tools will significantly transform and improve the field.

Outlines

00:00

🔬 From Physics to Social Science: A Journey

The speaker, originally a physicist, transitioned into sociology after studying chaos theory and social networks. Their diverse academic background across physics, sociology, and computer science provided them with a unique perspective on social science problems, which they believe differ in nature from traditional scientific challenges. Reflecting on their career path, the speaker introduces the idea that social science questions often appear simple, unlike those in fields like physics or chemistry.

05:01

💡 The Complexity of Social Science vs. Common Sense

The speaker explores the perception that social science problems seem simpler than those in hard sciences, yet social challenges remain unsolved. They highlight the gap between the ease of solving rocket science and the difficulty in addressing societal issues like government policies or financial crises. The speaker argues that this disparity arises because of our reliance on 'common sense,' which helps us navigate daily life but can mislead us in complex, large-scale social systems.

10:01

🤔 The Fallacy of 'Obvious' Explanations

Using historical examples, the speaker illustrates how common sense can mislead us into thinking certain outcomes are 'obvious.' The example of a World War II study, where results about city and rural soldiers were misinterpreted based on preconceived notions, showcases how we tend to rationalize outcomes after knowing them, even when they contradict our intuition. The speaker emphasizes the danger of labeling explanations as 'obvious' after the fact, pointing out that it undermines our understanding of the complexity involved in social science.

15:02

📚 Stories vs. Scientific Explanations

The speaker argues that while narratives help us make sense of historical events, they are not true explanations of causality. Stories help us understand the sequence of events, but they don't provide the 'why.' This leads to the common fallacy of believing that history repeats itself, even though the uniqueness of social systems means such repetition is rare. The speaker warns against using historical narratives to predict future events, as they can lead to unintended consequences.

Keywords

💡Chaos Theory

Chaos theory is a branch of mathematics focused on systems that are highly sensitive to initial conditions, often leading to seemingly random or unpredictable behavior. In the script, the speaker references Chaos Theory as a 'hot topic' during his undergraduate studies in physics, illustrating his early academic focus and how it contrasts with his current interest in social networks, which deal with complex systems in human interactions.

💡Social Networks

Social networks refer to the structures made up of individuals or organizations, which are connected through various forms of interactions such as friendship, work, or communication. The speaker's dissertation on social networks highlights the importance of understanding human interactions at scale, particularly how these interactions can be mapped and analyzed to understand social behavior, a topic that has gained increasing relevance.

💡Common Sense

Common sense is defined as the basic level of practical knowledge and reasoning that is shared by most people. In the video, the speaker critiques how common sense is often misapplied to complex social issues. He emphasizes that while common sense helps navigate everyday situations, it can be misleading when applied to large-scale, complex problems such as economic crises or societal trends, which require more rigorous scientific methods.

💡Rocket Science

Rocket science is used metaphorically to refer to a field that is perceived as extremely difficult and intellectually demanding. The speaker contrasts this perception with social science, noting that while rocket science problems like sending a satellite to Jupiter are solvable, social problems—such as measuring advertising effectiveness or predicting political crises—are far more elusive, despite being perceived as more 'obvious' or simpler.

💡Scientific Method

The scientific method is a systematic process used to investigate phenomena, acquire new knowledge, or correct and integrate previous knowledge. The speaker advocates for using the scientific method in social sciences, especially when common sense falls short. He argues that applying this method to social problems, particularly with the rise of data from digital interactions, can lead to more accurate insights and predictions.

💡Unintended Consequences

Unintended consequences refer to outcomes that are not foreseen or intended by a purposeful action. In the context of the video, the speaker warns that when we apply common sense or simplistic solutions to complex problems, we often face unintended consequences. He uses this to argue for a more systematic, data-driven approach to understanding and solving social issues.

💡Data Revolution

The data revolution refers to the vast increase in the amount of digital data generated by human activity, especially online. The speaker sees this as transformative for social science, likening the advent of big data to the invention of the telescope in astronomy. By analyzing data from platforms like Twitter and Facebook, social scientists can now observe and measure human interactions on a previously unimaginable scale.

💡Narratives

Narratives are stories or explanations that we create to make sense of events or phenomena. The speaker criticizes the human tendency to create narratives after the fact, especially when interpreting complex social events like financial crises. He argues that these narratives are often misleading, as they only seem obvious in hindsight and do not provide real explanatory power for predicting future events.

💡Complex Systems

Complex systems are systems made up of many interconnected parts, where the interactions between the parts produce behaviors that are difficult to predict. The speaker uses this concept to describe human societies and economies, where millions of people interact in unpredictable ways. He points out that common sense often fails in understanding complex systems because they do not behave like simpler, more predictable systems.

💡Social Science vs. Physical Science

The comparison between social science and physical science highlights the differences in studying human behavior versus natural phenomena. The speaker argues that while physical sciences like physics and rocket science deal with systems governed by precise laws, social sciences face much greater complexity and unpredictability, making them harder to analyze and predict with the same level of certainty.

Highlights

The speaker has a unique background, transitioning from physics to sociology and now works at Yahoo Research, blending disciplines like physics, mathematics, sociology, and computer science.

The speaker discusses the common perception that social science problems seem easier than physical science problems, but argues that social issues are often more complex.

The phrase 'This is not rocket science' is misleading, as solving social science problems, like measuring ad effectiveness or understanding foreign aid impact, is actually more challenging.

Social science deals with problems involving millions of people interacting in complex ways over time, which makes these problems inherently more difficult to solve.

The speaker suggests that common sense, while useful for everyday situations, fails when applied to large-scale, complex social problems.

An example is given from World War II research, where rural men were expected to perform better in the army, but the study showed city men actually fared better. This challenges the idea of obviousness and common sense.

The speaker highlights that narratives often mislead us into thinking we can predict future outcomes based on historical events, but history never truly repeats itself.

Common sense is powerful in guiding daily behavior but often fails in predicting outcomes in complex, large-scale scenarios like economic crises or marketing campaigns.

The scientific method can help solve social science problems where intuition fails, emphasizing the importance of systematic, evidence-based approaches.

The rise of digital technologies, such as the internet and social media, has allowed social scientists to observe and measure human interactions on a massive scale, which was not possible a decade ago.

The speaker compares the internet to a telescope for social science, enabling researchers to study social interactions in unprecedented ways.

Even with large-scale data and advanced tools, social science may never have the precise laws and formulas that characterize physics, due to the inherent complexity of human behavior.

The speaker is optimistic that the data revolution will significantly transform social science and influence fields like business and policy.

The speaker encourages a more scientific approach to addressing problems in policy, marketing, and strategy, suggesting that systematic methods will yield better results.

Despite the challenges, the speaker believes that using data-driven insights from digital platforms can offer new perspectives on solving complex social problems.

Transcripts

[Applause]

thanks Dave um so as Dave mentioned I'm

a

sociologist but uh if I can get my first

slide up here I am some of an unusual

kind of sociologist because I started

out life uh as a physicist uh back in

Australia I wrote my undergraduate uh

thesis on what was a very hot topic

called Chaos Theory uh a couple of years

later I then moved over here to the US

in 1993 and did my PhD in engineering at

Cornell University uh and I wrote my

dissertation then on a topic that was

not very hot at the time but has

subsequently become even hotter than

Chaos Theory namely social networks so

inspired by that experience and after a

couple of stints around the country

doing various postdoc fellowships in

Santa Fe and at MIT I ended up at

Columbia University teaching sociology

and I was there for about s years and

then just as I was getting comfortable

calling myself a soci ologist I moved

over to Yahoo research where I am now

and I've been there for the last four

years working almost exclusively with

computer scientists so the reason I'm

telling you this is that all this sort

of toing and frowing between the

disciplines from physics to mathematics

to Sociology to computer science has

really given me a sort of unusual

perspective on on social science itself

and in particular has led me to wonder

what it is about the problems that

sociologists study that make them seem

different to us than the the problems

that other kinds of scientists study so

to clarify this difference for you think

back to your undergraduate days or

possibly even your your high school days

and think about the courses that you

would have taken back then uh in

sociology in Psychology maybe even in

political science and you might you may

or may not have found these questions

interesting uh but I'm guessing that the

questions that you encountered in these

courses did not seem hard to you in the

way that the problems that you

encountered say in physics or in

chemistry or in biology seemed hard and

this attitude this difference between

the the this difference in difficulty

that we perceive between the problems of

sociology and the problems of science

manifests itself in this wonderful piffy

phrase that we often wheel out when

we're having trouble solving a problem

and we look around the room and we say

come on people we can do this this is

not rocket science now I find this

phrase as someone who did used to do

something a little bit like rocket

science and now does sociology very

puzzling because the evidence is that

we're extremely good at rocket science

okay so if NASA sends a a satellite to

Jupiter it actually goes where it's

supposed to go and it does what it's

supposed to do by contrast a 100 years

after John Waker said half the money I

spent on Advertising is wasted I just

don't know which half advertising

Executives still have tremendous

difficulty measuring the effectiveness

of their ad campaigns over 200 years

after the founding of the modern

Republic politicians still argue for

ferociously about the role of government

and whether or or even uh you know or

how it can solve the problems of

citizens over 300 years of Economic and

political Theory we still cannot predict

Financial or political crisis several

decades and trillions of dollars of of

of foreign aid and we still have

difficulty understanding how to help

people uh in the developing world even

at the more sort of prosaic level of

predicting the next hit book or hit

movie uh or even hit company uh experts

with lots of experience and lots of skin

in the game have tremendous difficulty

and after thousands of years of

educating our children we still argue

about how to do it so the evidence is

that social problems are extremely

difficulty and the question I want to

ask is why given all of this rocket

science seems hard and social science

seems like a matter of Common Sense and

the claim that I want to make today is

that it has to do with the nature of

Common Sense itself so what is common

sense well it's interesting this is a

term that we Bandy around a lot and it's

notoriously difficult to Define and in

fact you can tell that if you ask a

bunch of people what is common sense

they'll all tell you that they know the

answer and they'll all give you a

different answer okay so my answer uh

for you know the purpose of today is

that common sense is the kind of

intelligence that we rely on to navigate

concrete everyday situations so if I

look around the room I notice that none

of you have shine up in your swimwear

today and uh you know but the chances

are that when you got dressed this

morning morning you didn't sort of have

to think deeply about whether or not to

wear uh your bikini and you know the

reason why we can do this you know every

kind of uh decision that we make every

day when we get up we have to there's

all kinds of rules that we have to

follow uh in getting dressed and we

don't have to think about them because

it's just common sense uh Common Sense

also tells us how to behave uh in

different kinds of circumstances so the

way that I'm speaking to you now is very

different from the way you were speaking

to each other during the lunch break and

the chances are if I sort of walked into

one of your groups during lunch and uh

started speaking to you like this you

would think it was very strange right

that I was somebody who didn't know how

to interact socially that I lacked

common sense I live in New York and you

know we spend a lot of time on the

Subways in New York and you know many of

the times they're crowded and if someone

is sort of stacked up against you in the

Subway on on a crowded train it's

unpleasant but it's not a big deal if

the train is empty and somebody comes

and stands right next to you it's

absolutely bizarre or try walking into

an elevator and facing the Crow instead

of facing the door and see what happens

there are so many rules that we follow

without even knowing that we're

following them and we don't need to

think about it because that's what

common sense does even for very abstract

things like how to balance fairness and

reciprocity and financial transactions

or even personal transactions we just

know what we're supposed to know we know

what we're supposed to do and we don't

even know how because it's just common

sense okay so what's the problem if

common sense is so good at helping us

reason about human behavior why you know

how does it fail and the answer is that

it is so good at dealing with these

everyday concrete situations we try to

use it to reason about human behavior

even in situations that are not concrete

everyday situations so the kinds of

problems that I just mentioned like

managing the economy designing a

marketing campaign and so on these are

not concrete everyday situations these

are situations that involve thousands or

millions or even hundreds of millions of

people who are all very different from

each other who occupy very different uh

uh uh contexts and they're all

interacting with each other in very

complex ways over extended periods of

time and when we use common sense to

reason about these kinds of situations

it can mislead us that that is the

central claim it turns out that this is

a problem that sociologists have worried

about for a long time way back in

1947 Paul lazell the Great American

sociologist who also taught at Columbia

incidentally was writing this very

interesting review of a book called The

American Soldier so the American Soldier

was a study that was commissioned by the

US war department uh during World War II

and sociologists went into the US Army

and interviewed about 600,000 soldiers

and asked them what life in the Army was

like so lazes is reviewing the

publication of this report and he gives

the reader a sense of what it contains

by describing some of the results and

result number two that he describes is

that men from rural backgrounds in

general fared better than men from city

background grounds laville then steps

back from the review and imagines the

reader's response and the reader says oh

that's obvious right of course men from

rural backgrounds did better than men

from cities men from rural backgrounds

are used to living outside they're used

to sleeping on the ground they're used

to physical labor they used to getting

up with the sun you know why did I need

such a vast and expensive study to tell

me what I could have figured out on my

own just using my Common Sense good

point lazel says except that the answers

that I just gave you were the opposite

of what the report actually found it was

really City men not rural men who did

better in the Army now if he told you

that answer to begin with the correct

answer you also could have reconciled

that one with other things that you

thought you knew right so yes of course

men from city backrounds did better they

used to large vertically integrated

organizations they're used to showing up

every day in something like a uniform

they used to strict chains of command

rigid hours you know again that is

obvious to me eliz's Fel says exactly

and here's the problem when every

conclusion and its opposite appear

equally obvious once you know the

answer there is a problem with the

concept of

obviousness so fast forward 60 years

Lazar felt observation is also true of

much more Complex events so we think

about the unfolding possible crisis

happening in Europe now it's very

complicated there are lots and lots of

potentially relevant factors and lots of

potential out uh outcomes that we can

imagine everything from you know a lot

of fuss about nothing to a total

meltdown of the Euro zone so when we

think about things that are happening

now it they seem uh deeply complex they

seem deeply ambiguous but if we look

back at the last financial crisis we see

a very different picture we know how

that one ended and so we can sort of

sift through the mix of all the

potentially relevant things and we can

construct this narrative that leads

inevitably to the outcome that we have

obs ered of course now that we know that

we had a financial crisis it's obvious

to us that housing was in a bubble it's

obvious to us that the securitization of

mortgages was a real problem that

mortgage standards were too lacks and

it's obvious to us given all of these

factors that we were inevitably headed

for a disaster it seems like a

deterministic thing now I want to make

two points the first one is we can

always do this no matter what the

outcome is there's always so many things

that could have been relevant we can

always sort of go through the box and

pick out the ones that lead seemingly to

the answer that we have uh that we know

we now have and and that's Lazard F's

point but the second point is and the

more subtle one is we can only do this

once we know the outcome we cannot do it

at the time and so the outcome the the

result of all of this is that the things

that we call explanations the things

that we call uh that we look to to help

us understand The Complex events that

have happened in the world are really

just stories they're not explanations at

all they tell us what happens they tell

us the sequence of events that led to

where we are they do not tell us why

okay is this bad well not always stories

are powerful there are many uh reasons

that we use stories and they are uh you

know they're inspiring to us they help

us get out of bed in the morning uh you

know stories I'm not advocating that we

get rid of stories what I am saying

however is that they're so powerful that

we we the stories that we tell about

history are so powerful we're tempted to

generalize from them to make predictions

and if you don't think do that look at

this quote this is George santiana the

philosopher who says those who do not

learn from history are condemned to

repeat it he is explicitly saying you

should look at these narratives that we

have about history and you should

generalize from them to make predictions

about the future now again this is not

always bad think about common sense

think about where it works if we get to

experience the same situation over and

over and over again we really can learn

cause and effect we really can learn

what we're supposed to do but the whole

problem with complex syst

is that history never really repeats

itself we can convince ourselves that it

does but it never really does and so

when we try to use this sort of

Narrative Approach to history to make

predictions about the future to design

policy to design our marketing campaign

to make our next video go viral whatever

it is we're trying to do we always run

into this law of unintended

consequences okay so at this point many

of you are feeling depressed uh what's

the point I came here to get inspired

what is this guy doing you know there is

uh there is a bright side to all of this

that that that even though this can seem

encouraging once you get it like once

you realize that your intuition is not

as good as you think it is at least for

certain kinds of uh of of judgments you

have to start thinking of other things

that you can do so what can you do well

what I would point to is a long history

of scientific progress where the

scientific method has helped us to

understand the world precisely where we

do not have intuition that is really all

the scientific method is and the claim

that I want to make and what I want to

Advocate uh is that the kinds of

problems that I'm talking about here in

policy and marketing and strategy can

all benefit from a more systematic a

more scientifically driven approach and

you might say okay that's fine but if

that's so obvious why hasn't that been

done and I think there's a very good

answer is that the problems again the

problems that I'm talking about are

problems that involve large numbers of

people interacting with each other in

complex ways now his historically and I

really mean like up until about 10 years

ago it was impossible to observe these

interactions and it's very very hard to

do science when you can't observe things

it's very hard to do science when you

cannot measure the things that you're

interested in and what has changed in

the last 10 years or so and why it's so

exciting for people like me to be sort

of at the intersection of of Social and

computational science is that the

internet has really kind of uh you know

unveiled has really made the invisible

visible has really given us the ability

to measure the interactions between even

hundreds of millions of people uh in

real time and over extended periods of

time and it sort of maybe a little bit

fanal to say this but it feels to many

of us working in the field uh as if you

know social science has sort of stumbled

on our equivalent of the telescope the

the the device the technology that made

the previously invisible visible and

historically that has led uh to dramatic

improvements uh in science and so

there's a la list of things I can talk

about there's so many uh projects that

we're working on we use Twitter we use

Facebook we use Amazon's Mechanical Turk

we use email we use search logs

everything you do online everywhere you

go online everyone you talk to online

you know you are not just doing things

you are also generating this enormous

sort of web of interactions that help us

as social scientists understand the

world on this very large scale now I

don't know where this is going you know

it's very early days and even with all

of this data even with this scientific

approach that I'm advocating it's

probably not going to be the case that

social science is ever going to be

anything like physics with its beautiful

elegant laws that apply everywhere in

the universe the social world is just

messier than that and there's probably

nothing that we can or should try to do

about it nevertheless this revolution in

data I I really do believe this will

certainly transform social science it's

already transforming it uh and I hope

and you know one thing that I want to

try to advocate for is that this

revolution can also change the way we

think about these other problems in

business uh and even in policy thank you