MIN-LESSON 8b: (Power Laws) Q& A
Summary
TLDRIn this insightful lecture, the speaker addresses the prevalence of power laws in various aspects of life, such as wars and pandemics, which exhibit an alpha close to one-half, indicating heavy tails. The speaker criticizes the use of small sample sizes to make general claims, emphasizing the importance of avoiding sampling errors in scientific analysis. They also explore the concept of infinity in the context of power laws, explaining how it differs from finite observations and how it contributes to the unpredictability of large-scale events. The talk concludes with a discussion on the origins of power laws, including the Matthew effect and social contagion, and how constraints can alter their distribution.
Takeaways
- 📚 The speaker discusses the concept of power laws and their relation to events like wars and pandemics, emphasizing the importance of understanding their statistical properties.
- 🕊️ Stephen Pinker's view on the decline of violence is critiqued, with the speaker pointing out the limitations of drawing conclusions from a small sample size and the potential for sampling errors.
- ⏱️ Wars and pandemics are highlighted as having power law distributions with an alpha close to one half, indicating heavy tails and the possibility of extreme events.
- 🔍 The speaker emphasizes the need for caution when making statistical claims based on historical data, especially given the unreliability of historical records and the potential for exaggeration over time.
- 📉 The concept of 'infinite' in the context of power laws is explained, describing it as a situation where the mean or upper bound is not well-defined due to the nature of the data.
- 📈 Power laws are contrasted with Gaussian distributions, with the former being characterized by heavy tails and the latter by a more balanced distribution around the mean.
- 🌐 The speaker introduces the idea of maximum entropy distributions as a way to understand why power laws occur and under what conditions they might not.
- 💡 The 'Matthew effect' and 'preferential attachment' are mentioned as mechanisms that can lead to power law distributions, where successful entities become more successful over time.
- 🏪 An example of how power laws can emerge is given through the scenario of storefront allocations, where success leads to greater opportunities and further success.
- 📊 The speaker discusses the transformation of data to fit power law distributions, such as converting the maximum possible deaths in a conflict to an equivalent 'infinity' for statistical analysis.
- 🚫 The existence of constraints, like energy or biological limits, is presented as a reason why not all phenomena follow power law distributions, with human height being an example of a constrained variable.
Q & A
What is the main topic discussed in the transcript?
-The main topic discussed in the transcript is the concept of power laws, particularly in the context of wars, pandemics, and their statistical properties.
Who is Stephen Pinker and what is his claim regarding violence?
-Stephen Pinker is a journalist and author who wrote a book claiming that violence has declined over history, based on his analysis of 100 observations.
What is the issue with using a small sample size to derive general properties, according to the speaker?
-The issue with using a small sample size is that it introduces sampling errors and the risk of being misled by randomness, which goes against the scientific method of avoiding noise and not being fooled by randomness.
What is the significance of the 'alpha' value in the context of power laws?
-The 'alpha' value in power laws represents the steepness of the tail of the distribution. Wars and pandemics have an alpha close to one half, indicating a thick tail, which means rare but extreme events are more likely than what a normal distribution would predict.
What is the average inter-arrival time for a conflict causing more than 50 million deaths, according to the speaker?
-The average inter-arrival time for a conflict causing more than 50 million deaths is approximately 80 years.
Why does the speaker argue that it is not scientifically valid to claim the world is a 'better place' without wars based on a short period of observation?
-The speaker argues that to make a statistically significant claim about the absence of large-scale wars, one must wait for a period at least three times the average inter-arrival time to account for the variability in such rare events.
What is the issue with historical data when it comes to statistical inference?
-The issue with historical data is that historians are not rigorous scientists; they often cite other people and provide numbers without a clear methodology, leading to potential inaccuracies in statistical inference.
How does the speaker address the problem of variability in historical death tolls due to conflicts?
-The speaker addresses this by building over 500 events with high and low estimates of death tolls, creating 100,000 different histories to test hypotheses and account for variability.
What is the concept of 'infinite' in the context of power laws and statistical analysis?
-In the context of power laws, 'infinite' means that the mean or other metrics do not converge to a specific value but rather fluctuate widely, indicating that the upper bound is not defined.
What are the two types of distributions mentioned by the speaker, and how do they relate to power laws?
-The two types of distributions mentioned are one-tailed distributions, which are skewed to the left or right, and two-tailed distributions, which have both positive and negative tails. Power laws are typically one-tailed, with a heavy emphasis on the right side (positive values).
How does the speaker explain the formation of power laws in real-world phenomena?
-The speaker explains the formation of power laws through mechanisms like the Matthew effect (rich get richer), preferential attachment, and social contagion effects, which can lead to a snowball effect and the creation of power-law distributions.
Why does the speaker believe that the world is naturally power law distributed, except under certain constraints?
-The speaker believes that the world is naturally power law distributed because many phenomena follow this pattern unless there are specific constraints like growth limitations, energy constraints, or biological constraints that prevent a power law distribution.
Outlines
📚 Questioning the Decline of Violence and Wars
The speaker addresses the question about the decline in violence and wars, referencing Stephen Pinker's book that suggests a decrease in violence over time. The speaker challenges this view by discussing the power law distribution of wars and pandemics, noting that they have a significant 'alpha' value close to one half, indicating a fat tail and the potential for large-scale events. The speaker criticizes the scientific approach of drawing conclusions from a small sample size and emphasizes the importance of avoiding sampling errors and not being misled by randomness. They also discuss the statistical significance of making claims about the absence of large-scale wars, suggesting that a longer time frame is necessary for such assertions. Additionally, the speaker touches on the unreliability of historical data, particularly when it comes to the numbers reported by historians, and proposes a method of building multiple histories to test hypotheses and account for this variability.
🔍 Understanding Power Laws and Infinite Observations
This paragraph delves into the concept of power laws, particularly focusing on the idea of infinity in the context of observations. The speaker explains that infinite in this sense does not mean an actual infinite observation but rather a theoretical construct that helps in understanding the distribution of data. They clarify that power laws are characterized by a single tail, which is either skewed to the left or right, and that variables of interest, such as wars and pandemics, typically have one tail. The speaker also discusses the transformation of data to accommodate the concept of infinity, allowing for the application of power laws in statistical inference. The paragraph concludes with a brief mention of the origins of power laws, hinting at a deeper exploration in the context of entropy and maximum entropy distributions.
🌐 The Emergence of Power Laws in Social Phenomena
The speaker explores the emergence of power laws in social phenomena, such as the 'Matthew effect' where the rich get richer, and the concept of preferential attachment. They provide an illustrative example of how a store with more visitors is likely to expand and attract even more visitors, leading to a power law distribution over time. The speaker also mentions social contagion effects, where the actions of a few can influence the behavior of many, further reinforcing the power law distribution. They contrast this with situations where constraints, such as energy or biological limitations, prevent the formation of power laws, as seen in the distribution of human height. The speaker concludes by emphasizing that power laws are a natural distribution in the absence of constraints, and they express a personal belief in the prevalence of power law distributions in the world.
Mindmap
Keywords
💡Power Laws
💡Stephen Pinker
💡Alpha
💡Black Swan
💡Sampling Error
💡Statistical Significance
💡Bootstrap Method
💡Existential Risk
💡Infinite
💡Matthew Effect
💡Constraints
Highlights
Stephen Pinker's book argues that violence has declined historically, but the speaker challenges this view with a power law perspective.
Wars and pandemics follow a power law with an alpha close to one half, indicating a thick tail of extreme events.
The critique of deriving general properties from a small sample size due to sampling errors and the importance of avoiding being fooled by randomness.
The average inter-arrival time for wars causing more than 50 million deaths is approximately 80 years, questioning the premature conclusion of a safer world.
The importance of waiting three times the average inter-arrival time for statistically significant claims about the absence of large-scale wars.
Historians' lack of rigor in data reporting and the issue of inflated numbers over time, affecting statistical inference.
The method of building over 500 alternate histories to account for the variability in historical conflict data.
The concept of transforming a finite number of deaths into an equivalent to infinity for power law analysis.
Pandemics also follow a power law distribution, reinforcing the existential risks they pose to humanity.
The clarification of the concept of infinity in data analysis, relating to undefined means and the unpredictability of extreme values.
The explanation of power laws as one-tailed distributions, typically skewed to the right, with implications for variables like wars and pandemics.
The role of constraints in preventing the formation of power laws, such as biological constraints on human height.
The Matthew effect and preferential attachment as mechanisms leading to power law distributions in social and economic phenomena.
The use of maximum entropy distribution to understand the formation of power laws and the constraints that lead to Gaussian distributions.
The speaker's personal view that the world is naturally power law distributed except when constrained, contrasting with the idea of social contagion effects.
The impact of constraints on growth, energy, and limitations in shaping distributions that deviate from power laws.
A commitment to answering more questions in the future, indicating an ongoing dialogue on the topic.
Transcripts
friends hello again uh
i'm gonna uh answer some questions
uh a lot of questions online about the
lecture on power laws
and thank you for these interesting
questions so let me start but i'm going
to structure it
to make it flow with the previous
[Music]
the main power law
lesson okay so
first question was
what about wars okay stephen pinker
wrote a book
the journalist stephen pinker wrote a
book on
a saying that violence has declined
naively he took 100 observations
uh over the past history and uh
claimed that you know we haven't had
wars since
big wars since the second world war so
the world is a better place
there were like the number of laws
in his book is enormous but i'll focus
on
one thing wars you remember i was
talking about alphas
wars have an alpha close to one half
the lower the alpha the
fatter the tail pandemics
also about one half and when i wrote the
black swan
that's what shocked me wars and
pandemics
they had the thickest tails and it was
you know
uh not until quite a bit later that we
did work on words and pandemics
in a more formal way so to give you the
intuition of why
uh what what error my picture i'm going
to focus on which is the one the most
offensive i find scientifically
which is to derive general properties of
a small sample
because you have sampling errors so the
whole idea about science is avoiding
noise
sampling error and and not being fooled
by randomness
so i will um
and i'll show you what what what in fact
had happened here okay when we read
the data we know that we did the
following
and actually the paper is in my book
okay or a version of the papers in my
book
you have you take inter-arrival time you
take an event
say an event
x higher than okay
you take an event say
x higher than
50 million in today's population the
equivalent 50 more than 50 million
have died in that event we call
a conflict single conflict not series of
conflicts
cumulatively killing more than 50
million and
mean time for it to happen
throughout history effectively for x
higher than 50 million
the mean time is something like uh
80 years
okay so if it takes 80 years on average
for such an event to happen
how can you make claims seven years
later
that you know the world is a better
place
we haven't had such a war you need to
wait
let's assume in standard deviation
actually this is gaussian the arrival
time
between events is gaussian this is time
you have wars
you put a bench 50 million how many
times you exceeded
the arrival time is gaussian and
memoryless and not gaussian sort of like
thin tailed but memoryless anyway so you
have to wait three times
that to start making a statistically
significant claim
this is not science so another problem
with historians with data coming from
history particularly when you go back
two thousand more than two thousand
years
that historians are not very rigorous
scientists
they just cite other people and give you
a number
so how can we have a
statistical inference that takes into
account the fact that historians are
literally bullshitters as follows
we take every conflict you have
this is a time you have a conflict you
take two numbers
low and high because visibly
let's say take the recent recent events
in the algerian war
you have low number what the french
think
and high number what algerians claim
and of course you have inflation over
time the french things three hundred
thousand say
were killed by the event and the
algerians saw that a million were killed
and of course you have inflation like
the hama massacre in
syria had inflation started at 2000
and reached 40 000 with no information
just like you relay
the event the number gets bigger so this
phenomenon so how can we know
if something was reliable in the past so
with this
what we do it did is built
hundreds of over 500 events a little
more than 500 events
you can build 100k histories
you build 100 000 histories where you
take
this high that low
that low high low so you build your
history
and and from there you test
your hypothesis and sure enough
practically all sample path
throughout history
all sample path
had a alpha below one
all second problem
is of course you know the variable the
random variable for them to be power law
they should reach infinity
we did the trick because you have a
finite number of people on the planet
to transform a hundred percent of the
population being killed into something
equivalent to infinity and that
transformation
doesn't change the numbers for what we
saw but just allows us to do power law
because it changes the numbers say if
you have more than eight billion
people dying or more than seven billion
yeah you see a difference in numbers
so it is uh what we call a log
transformation
and and that allows us to use power laws
quite effectively
in doing the inference so
now i answered the question about wars
let me talk about pandemics same methods
we use from pandemics
we catalog practically all pandemics
recorded
and figure that there's a power law and
then of course you bootstrap
you jack now if you remove some
observation add uh you know
whatever you do get the same result
uh both have an alpha below one
so both represent existential
risks for us and they're both very
dangerous
so i've answered
the first question about war and
pandemics
second point someone was saying well
what do you mean by infinite
i never observe infinite mean you never
have finite
infinite observation it's always finite
let me clean this
infinite mean means that you run
your uh your data
and every run will provide you with a
different mean
and they're all going to be high you
don't know what the upper bound is
so that's one one way to view it another
one
is by building sums i have numbers going
from x1
to whatever x and and m you know you
make that large as you want and i take s
n sigma x i one over n so
the average and you start here
if the thing converges it'll be volatile
and then it reaches a line if your mean
is
undefined it'll be all over the map
typically the terminology is as follows
where you say
undefined or infinite
if the metric you're looking for say the
mean
is between zero and infinity you call it
infinite if it's between minus infinity
and plus infinity you're caused
undefined because it could be plus
infinity or minus infinity or anywhere
in between
so power laws and one-tailed
have an alpha here but not here
okay this is sort of like skewed left
right distribution
you can have distribution like that or
again distribution with both tails
this is the tails detail for most
variables we're concerned with
we have one tail like wars one tail
pandemics one tail
you don't worry about negative things
and finance when we talk about returns
log returns can be negative infinity
plus infinity
something going to zero is negative
infinity in long returns
sort of the same game we played with uh
with
pandemics so you have a left tail
and you have a right tail
where where do power laws come from
when we do entropy we're going to look
at a minimum
uh sorry a maximum entropy distribution
and then we'll we'll have a more formal
understanding of it
and that's the constraints that make
things a gaussian
not uh it's like we start with the
gaussian things become power law no
if you put constraints of energy on
anything you have a finite variance and
that sort of like
allows you to use the central limit
because you're bounded in the variance
but but let me
give you a story of how things become
paralleled and
also why they don't stay there the
literature has
what's called the matthew effects with
rich get richer
or something called preferential
attachment and the third experiment is
as follows
you have n people
with one over and a location of store
storefront
okay we have a storefront each one there
they all have equal storefront
that's day zero day one you have
customers
visiting these p randomly making
uniform so and let's name them x1
so x3 xyz so x3
has more visitors than the others it's
going to have a larger storefront
so this is day zero if on day one
you now reallocate storefronts because
x3 has
made some money so she or he you will
you know expand
so when you expand your probability of
getting visitors is no longer one over n
it's gonna be whenever you're
you're higher than one over n and the
other would have lower probability one
over n
and and and after over time the big will
get bigger
because you have a higher probability of
making money until the thing collapses
that's sort of the story another way to
view it
in saying social
contagion effects is that's all i go to
a store
and i see a friend
buying a book okay
that person may have randomly bought
this book
so i go by the book all right
now a third person will see two people
walking around the same book
what's going on i'll buy this book and
then sure enough it causes a
snowball effect and you'll have a power
law
i personally don't like these
representation i think that the world
is naturally power law distributed
except when you have constraints
constraints of growth constraints of
energy constraints of limitations that
you have
for example a price and the market
doesn't have constraints well the gdp
is very far away so therefore
it can follow a power law however height
there are energy constraints or the
height constraints are biological
constraints that makes human
not follow power law and the
distribution of height
although between species we tend to have
parallel like think of a mouse
versus an elephant so
thank you i'll answer more questions
later i just wanted to keep this short
have a great day bye now
5.0 / 5 (0 votes)