MIN-LESSON 8b: (Power Laws) Q& A

Summary

In this insightful lecture, the speaker addresses the prevalence of power laws in various aspects of life, such as wars and pandemics, which exhibit an alpha close to one-half, indicating heavy tails. The speaker criticizes the use of small sample sizes to make general claims, emphasizing the importance of avoiding sampling errors in scientific analysis. They also explore the concept of infinity in the context of power laws, explaining how it differs from finite observations and how it contributes to the unpredictability of large-scale events. The talk concludes with a discussion on the origins of power laws, including the Matthew effect and social contagion, and how constraints can alter their distribution.

Takeaways

• 📚 The speaker discusses the concept of power laws and their relation to events like wars and pandemics, emphasizing the importance of understanding their statistical properties.
• 🕊️ Stephen Pinker's view on the decline of violence is critiqued, with the speaker pointing out the limitations of drawing conclusions from a small sample size and the potential for sampling errors.
• ⏱️ Wars and pandemics are highlighted as having power law distributions with an alpha close to one half, indicating heavy tails and the possibility of extreme events.
• 🔍 The speaker emphasizes the need for caution when making statistical claims based on historical data, especially given the unreliability of historical records and the potential for exaggeration over time.
• 📉 The concept of 'infinite' in the context of power laws is explained, describing it as a situation where the mean or upper bound is not well-defined due to the nature of the data.
• 📈 Power laws are contrasted with Gaussian distributions, with the former being characterized by heavy tails and the latter by a more balanced distribution around the mean.
• 🌐 The speaker introduces the idea of maximum entropy distributions as a way to understand why power laws occur and under what conditions they might not.
• 💡 The 'Matthew effect' and 'preferential attachment' are mentioned as mechanisms that can lead to power law distributions, where successful entities become more successful over time.
• 🏪 An example of how power laws can emerge is given through the scenario of storefront allocations, where success leads to greater opportunities and further success.
• 📊 The speaker discusses the transformation of data to fit power law distributions, such as converting the maximum possible deaths in a conflict to an equivalent 'infinity' for statistical analysis.
• 🚫 The existence of constraints, like energy or biological limits, is presented as a reason why not all phenomena follow power law distributions, with human height being an example of a constrained variable.

Q & A

What is the main topic discussed in the transcript?

The main topic discussed in the transcript is the concept of power laws, particularly in the context of wars, pandemics, and their statistical properties.

Who is Stephen Pinker and what is his claim regarding violence?

Stephen Pinker is a journalist and author who wrote a book claiming that violence has declined over history, based on his analysis of 100 observations.

What is the issue with using a small sample size to derive general properties, according to the speaker?

The issue with using a small sample size is that it introduces sampling errors and the risk of being misled by randomness, which goes against the scientific method of avoiding noise and not being fooled by randomness.

What is the significance of the 'alpha' value in the context of power laws?

The 'alpha' value in power laws represents the steepness of the tail of the distribution. Wars and pandemics have an alpha close to one half, indicating a thick tail, which means rare but extreme events are more likely than what a normal distribution would predict.

What is the average inter-arrival time for a conflict causing more than 50 million deaths, according to the speaker?

The average inter-arrival time for a conflict causing more than 50 million deaths is approximately 80 years.

Why does the speaker argue that it is not scientifically valid to claim the world is a 'better place' without wars based on a short period of observation?

The speaker argues that to make a statistically significant claim about the absence of large-scale wars, one must wait for a period at least three times the average inter-arrival time to account for the variability in such rare events.

What is the issue with historical data when it comes to statistical inference?

The issue with historical data is that historians are not rigorous scientists; they often cite other people and provide numbers without a clear methodology, leading to potential inaccuracies in statistical inference.

How does the speaker address the problem of variability in historical death tolls due to conflicts?

The speaker addresses this by building over 500 events with high and low estimates of death tolls, creating 100,000 different histories to test hypotheses and account for variability.

What is the concept of 'infinite' in the context of power laws and statistical analysis?

In the context of power laws, 'infinite' means that the mean or other metrics do not converge to a specific value but rather fluctuate widely, indicating that the upper bound is not defined.

What are the two types of distributions mentioned by the speaker, and how do they relate to power laws?

The two types of distributions mentioned are one-tailed distributions, which are skewed to the left or right, and two-tailed distributions, which have both positive and negative tails. Power laws are typically one-tailed, with a heavy emphasis on the right side (positive values).

How does the speaker explain the formation of power laws in real-world phenomena?

The speaker explains the formation of power laws through mechanisms like the Matthew effect (rich get richer), preferential attachment, and social contagion effects, which can lead to a snowball effect and the creation of power-law distributions.

Why does the speaker believe that the world is naturally power law distributed, except under certain constraints?

The speaker believes that the world is naturally power law distributed because many phenomena follow this pattern unless there are specific constraints like growth limitations, energy constraints, or biological constraints that prevent a power law distribution.

Outlines

00:00

📚 Questioning the Decline of Violence and Wars

The speaker addresses the question about the decline in violence and wars, referencing Stephen Pinker's book that suggests a decrease in violence over time. The speaker challenges this view by discussing the power law distribution of wars and pandemics, noting that they have a significant 'alpha' value close to one half, indicating a fat tail and the potential for large-scale events. The speaker criticizes the scientific approach of drawing conclusions from a small sample size and emphasizes the importance of avoiding sampling errors and not being misled by randomness. They also discuss the statistical significance of making claims about the absence of large-scale wars, suggesting that a longer time frame is necessary for such assertions. Additionally, the speaker touches on the unreliability of historical data, particularly when it comes to the numbers reported by historians, and proposes a method of building multiple histories to test hypotheses and account for this variability.

05:02

🔍 Understanding Power Laws and Infinite Observations

This paragraph delves into the concept of power laws, particularly focusing on the idea of infinity in the context of observations. The speaker explains that infinite in this sense does not mean an actual infinite observation but rather a theoretical construct that helps in understanding the distribution of data. They clarify that power laws are characterized by a single tail, which is either skewed to the left or right, and that variables of interest, such as wars and pandemics, typically have one tail. The speaker also discusses the transformation of data to accommodate the concept of infinity, allowing for the application of power laws in statistical inference. The paragraph concludes with a brief mention of the origins of power laws, hinting at a deeper exploration in the context of entropy and maximum entropy distributions.

10:03

🌐 The Emergence of Power Laws in Social Phenomena

The speaker explores the emergence of power laws in social phenomena, such as the 'Matthew effect' where the rich get richer, and the concept of preferential attachment. They provide an illustrative example of how a store with more visitors is likely to expand and attract even more visitors, leading to a power law distribution over time. The speaker also mentions social contagion effects, where the actions of a few can influence the behavior of many, further reinforcing the power law distribution. They contrast this with situations where constraints, such as energy or biological limitations, prevent the formation of power laws, as seen in the distribution of human height. The speaker concludes by emphasizing that power laws are a natural distribution in the absence of constraints, and they express a personal belief in the prevalence of power law distributions in the world.

Keywords

💡Power Laws

Power laws are mathematical relationships between two quantities, where a relative change in one quantity results in a proportional relative change in the other quantity, independent of the initial size of those quantities. In the context of the video, power laws are discussed in relation to the distribution of events like wars and pandemics, where the frequency of such events follows a specific mathematical pattern. The speaker uses power laws to analyze the likelihood and impact of rare but extreme events, such as large-scale wars that result in a significant number of deaths.

💡Stephen Pinker

Stephen Pinker is a cognitive scientist, psychologist, and author known for his books on language, mind, and human nature. In the video, the speaker refers to Pinker's work on the decline of violence, challenging the idea that the world is becoming a better place by pointing out the limitations of Pinker's methodology and the potential for large-scale conflicts that could contradict this narrative.

💡Alpha

In the context of power laws, the 'alpha' refers to the exponent in the mathematical formula that describes the relationship between two quantities. A lower alpha value indicates a 'fatter' tail in the distribution, meaning that there is a higher likelihood of extreme events occurring. The speaker mentions that wars and pandemics have an alpha close to one half, suggesting that these events have a significant potential for large-scale impact.

💡Black Swan

The term 'Black Swan' is used to describe an event that is highly improbable but has severe consequences and is typically only understood in hindsight. The speaker refers to their work on the 'Black Swan' to highlight the unpredictability of events like wars and pandemics, which have thick tails in their distribution, indicating a higher likelihood of extreme outcomes.

💡Sampling Error

Sampling error refers to the error that arises from the difference between the characteristics of the sample and the population from which it is drawn. In the video, the speaker criticizes the use of small sample sizes to make broad claims about trends, such as the decline of violence, emphasizing the importance of avoiding sampling errors to prevent being misled by randomness.

💡Statistical Significance

Statistical significance is a measure of how likely it is that an observed effect or difference occurred by chance. The speaker argues that to make a statistically significant claim about the decline of wars, one must wait a sufficient amount of time, which is three times the average inter-arrival time of such events, to ensure that the claim is not a result of random variation.

💡Bootstrap Method

The bootstrap method is a resampling technique used in statistics to estimate the accuracy of sample estimates. It involves resampling the data multiple times and calculating the statistic of interest for each resample. In the video, the speaker mentions using the bootstrap method to analyze data on pandemics, suggesting that the power law distribution observed holds even when different subsets of the data are considered.

💡Existential Risk

Existential risk refers to a risk that threatens the entire future of humanity, potentially leading to human extinction or a permanent and drastic reduction in the potential for future development. The speaker identifies wars and pandemics as existential risks due to their potential to cause extreme harm and disruption on a global scale.

💡Infinite

In the context of the video, 'infinite' is used to describe a situation where the mean or sum of a dataset does not converge to a specific value but instead varies widely, indicating a lack of an upper bound. The speaker explains that in power law distributions, the concept of 'infinite' is used to illustrate the potential for extreme values that can significantly impact the overall distribution.

💡Matthew Effect

The Matthew Effect, or 'rich get richer' phenomenon, is a principle from sociology that suggests that those who have a certain advantage are more likely to accumulate additional advantages. In the video, the speaker uses this concept to explain how power laws can emerge, as those with more resources or success are more likely to gain even more, leading to a distribution where a few entities dominate.

💡Constraints

Constraints refer to limitations or restrictions that can affect the behavior of a system or the distribution of certain variables. The speaker discusses how constraints, such as energy limitations or biological constraints on human height, can prevent certain variables from following a power law distribution. In contrast, variables like GDP or market prices, which are less constrained, can exhibit power law characteristics.

Highlights

Stephen Pinker's book argues that violence has declined historically, but the speaker challenges this view with a power law perspective.

Wars and pandemics follow a power law with an alpha close to one half, indicating a thick tail of extreme events.

The critique of deriving general properties from a small sample size due to sampling errors and the importance of avoiding being fooled by randomness.

The average inter-arrival time for wars causing more than 50 million deaths is approximately 80 years, questioning the premature conclusion of a safer world.

The importance of waiting three times the average inter-arrival time for statistically significant claims about the absence of large-scale wars.

Historians' lack of rigor in data reporting and the issue of inflated numbers over time, affecting statistical inference.

The method of building over 500 alternate histories to account for the variability in historical conflict data.

The concept of transforming a finite number of deaths into an equivalent to infinity for power law analysis.

Pandemics also follow a power law distribution, reinforcing the existential risks they pose to humanity.

The clarification of the concept of infinity in data analysis, relating to undefined means and the unpredictability of extreme values.

The explanation of power laws as one-tailed distributions, typically skewed to the right, with implications for variables like wars and pandemics.

The role of constraints in preventing the formation of power laws, such as biological constraints on human height.

The Matthew effect and preferential attachment as mechanisms leading to power law distributions in social and economic phenomena.

The use of maximum entropy distribution to understand the formation of power laws and the constraints that lead to Gaussian distributions.

The speaker's personal view that the world is naturally power law distributed except when constrained, contrasting with the idea of social contagion effects.

The impact of constraints on growth, energy, and limitations in shaping distributions that deviate from power laws.

A commitment to answering more questions in the future, indicating an ongoing dialogue on the topic.

Transcripts

friends hello again uh

i'm gonna uh answer some questions

uh a lot of questions online about the

lecture on power laws

and thank you for these interesting

questions so let me start but i'm going

to structure it

to make it flow with the previous

[Music]

the main power law

lesson okay so

first question was

what about wars okay stephen pinker

wrote a book

the journalist stephen pinker wrote a

book on

a saying that violence has declined

naively he took 100 observations

uh over the past history and uh

claimed that you know we haven't had

wars since

big wars since the second world war so

the world is a better place

there were like the number of laws

in his book is enormous but i'll focus

on

one thing wars you remember i was

talking about alphas

wars have an alpha close to one half

the lower the alpha the

fatter the tail pandemics

also about one half and when i wrote the

black swan

that's what shocked me wars and

pandemics

they had the thickest tails and it was

you know

uh not until quite a bit later that we

did work on words and pandemics

in a more formal way so to give you the

intuition of why

uh what what error my picture i'm going

to focus on which is the one the most

offensive i find scientifically

which is to derive general properties of

a small sample

because you have sampling errors so the

whole idea about science is avoiding

noise

sampling error and and not being fooled

by randomness

so i will um

and i'll show you what what what in fact

had happened here okay when we read

the data we know that we did the

following

and actually the paper is in my book

okay or a version of the papers in my

book

you have you take inter-arrival time you

take an event

say an event

x higher than okay

you take an event say

x higher than

50 million in today's population the

equivalent 50 more than 50 million

have died in that event we call

a conflict single conflict not series of

conflicts

cumulatively killing more than 50

million and

mean time for it to happen

throughout history effectively for x

higher than 50 million

the mean time is something like uh

80 years

okay so if it takes 80 years on average

for such an event to happen

how can you make claims seven years

later

that you know the world is a better

place

we haven't had such a war you need to

wait

let's assume in standard deviation

actually this is gaussian the arrival

time

between events is gaussian this is time

you have wars

you put a bench 50 million how many

times you exceeded

the arrival time is gaussian and

memoryless and not gaussian sort of like

thin tailed but memoryless anyway so you

have to wait three times

that to start making a statistically

significant claim

this is not science so another problem

with historians with data coming from

history particularly when you go back

two thousand more than two thousand

years

that historians are not very rigorous

scientists

they just cite other people and give you

a number

so how can we have a

statistical inference that takes into

account the fact that historians are

literally bullshitters as follows

we take every conflict you have

this is a time you have a conflict you

take two numbers

low and high because visibly

let's say take the recent recent events

in the algerian war

you have low number what the french

think

and high number what algerians claim

and of course you have inflation over

time the french things three hundred

thousand say

were killed by the event and the

algerians saw that a million were killed

and of course you have inflation like

the hama massacre in

syria had inflation started at 2000

and reached 40 000 with no information

just like you relay

the event the number gets bigger so this

phenomenon so how can we know

if something was reliable in the past so

with this

what we do it did is built

hundreds of over 500 events a little

more than 500 events

you can build 100k histories

you build 100 000 histories where you

take

this high that low

that low high low so you build your

history

and and from there you test

your hypothesis and sure enough

practically all sample path

throughout history

all sample path

had a alpha below one

all second problem

is of course you know the variable the

random variable for them to be power law

they should reach infinity

we did the trick because you have a

finite number of people on the planet

to transform a hundred percent of the

population being killed into something

equivalent to infinity and that

transformation

doesn't change the numbers for what we

saw but just allows us to do power law

because it changes the numbers say if

you have more than eight billion

people dying or more than seven billion

yeah you see a difference in numbers

so it is uh what we call a log

transformation

and and that allows us to use power laws

quite effectively

in doing the inference so

now i answered the question about wars

let me talk about pandemics same methods

we use from pandemics

we catalog practically all pandemics

recorded

and figure that there's a power law and

then of course you bootstrap

you jack now if you remove some

observation add uh you know

whatever you do get the same result

uh both have an alpha below one

so both represent existential

risks for us and they're both very

dangerous

so i've answered

the first question about war and

pandemics

second point someone was saying well

what do you mean by infinite

i never observe infinite mean you never

have finite

infinite observation it's always finite

let me clean this

infinite mean means that you run

your uh your data

and every run will provide you with a

different mean

and they're all going to be high you

don't know what the upper bound is

so that's one one way to view it another

one

is by building sums i have numbers going

from x1

to whatever x and and m you know you

make that large as you want and i take s

n sigma x i one over n so

the average and you start here

if the thing converges it'll be volatile

and then it reaches a line if your mean

is

undefined it'll be all over the map

typically the terminology is as follows

where you say

undefined or infinite

if the metric you're looking for say the

mean

is between zero and infinity you call it

infinite if it's between minus infinity

and plus infinity you're caused

undefined because it could be plus

infinity or minus infinity or anywhere

in between

so power laws and one-tailed

have an alpha here but not here

okay this is sort of like skewed left

right distribution

you can have distribution like that or

again distribution with both tails

this is the tails detail for most

variables we're concerned with

we have one tail like wars one tail

pandemics one tail

you don't worry about negative things

and finance when we talk about returns

log returns can be negative infinity

plus infinity

something going to zero is negative

infinity in long returns

sort of the same game we played with uh

with

pandemics so you have a left tail

and you have a right tail

where where do power laws come from

when we do entropy we're going to look

at a minimum

uh sorry a maximum entropy distribution

and then we'll we'll have a more formal

understanding of it

and that's the constraints that make

things a gaussian

not uh it's like we start with the

gaussian things become power law no

if you put constraints of energy on

anything you have a finite variance and

that sort of like

allows you to use the central limit

because you're bounded in the variance

but but let me

give you a story of how things become

paralleled and

also why they don't stay there the

literature has

what's called the matthew effects with

rich get richer

or something called preferential

attachment and the third experiment is

as follows

you have n people

with one over and a location of store

storefront

okay we have a storefront each one there

they all have equal storefront

that's day zero day one you have

customers

visiting these p randomly making

uniform so and let's name them x1

so x3 xyz so x3

has more visitors than the others it's

going to have a larger storefront

so this is day zero if on day one

you now reallocate storefronts because

x3 has

made some money so she or he you will

you know expand

so when you expand your probability of

getting visitors is no longer one over n

it's gonna be whenever you're

you're higher than one over n and the

other would have lower probability one

over n

and and and after over time the big will

get bigger

because you have a higher probability of

making money until the thing collapses

that's sort of the story another way to

view it

in saying social

contagion effects is that's all i go to

a store

and i see a friend

buying a book okay

that person may have randomly bought

this book

so i go by the book all right

now a third person will see two people

walking around the same book

what's going on i'll buy this book and

then sure enough it causes a

snowball effect and you'll have a power

law

i personally don't like these

representation i think that the world

is naturally power law distributed

except when you have constraints

constraints of growth constraints of

energy constraints of limitations that

you have

for example a price and the market

doesn't have constraints well the gdp

is very far away so therefore

it can follow a power law however height

there are energy constraints or the

height constraints are biological

constraints that makes human

not follow power law and the

distribution of height

although between species we tend to have

parallel like think of a mouse

versus an elephant so

thank you i'll answer more questions

later i just wanted to keep this short

have a great day bye now