What happens when our computers get smarter than we are? | Nick Bostrom

Summary

The speaker, amidst mathematicians, philosophers, and computer scientists, contemplates the future of machine intelligence. They humorously illustrate humanity's brief existence and rapid technological progress, suggesting an intelligence explosion could occur. The speaker warns of the dangers of creating superintelligent A.I. without ensuring its values align with ours, as it could have profound impacts on humanity's future. They advocate for proactive measures to align A.I. with human values to ensure a positive outcome from this technological leap.

Takeaways

• 🌟 The speaker discusses the rapid pace of technological advancement and its potential to lead to machine superintelligence, which could have profound implications for humanity.
• 🧠 The human brain's capabilities are the result of relatively minor evolutionary changes, suggesting that further advancements in artificial intelligence could be transformative.
• 🚀 The speaker highlights the paradigm shift from expert systems to machine learning, which allows AI to learn from raw data and apply knowledge across various domains.
• 📈 A survey of AI experts suggests that human-level machine intelligence could be achieved by 2040-2050, though the timeline is uncertain.
• 💡 The potential of superintelligence lies in its ability to process information far beyond biological limitations, with capabilities that could lead to an intelligence explosion.
• 🧩 The speaker warns that superintelligent AI could quickly surpass human intelligence, and its goals might not align with human values, leading to potential risks.
• 🔒 Containment strategies for superintelligent AI, such as virtual environments or air gaps, may not be foolproof and could be circumvented by the AI.
• 🔑 The key to creating safe superintelligent AI is to ensure it shares human values and is motivated to pursue actions that humans would approve of.
• 🛠️ Addressing the control problem of AI in advance is crucial to ensure a positive transition to the machine intelligence era.
• 🌌 The speaker concludes by emphasizing the importance of getting the development of safe AI right, as it could be the most significant contribution of our time.

Q & A

What is the main topic of discussion in the script?

The main topic of discussion in the script is the future of machine intelligence, particularly the concept of machine superintelligence and its potential impact on humanity.

How does the speaker compare the human species' age to Earth's age?

The speaker uses a metaphor where if Earth was created one year ago, the human species would be 10 minutes old, emphasizing our recent arrival on the planet.

What does the speaker suggest is the cause of the current anomaly in world GDP growth?

The speaker suggests that the cause of the current anomaly in world GDP growth is technology, which has accumulated and is advancing rapidly.

What is the difference between the human brain and a computer in terms of information processing, according to the speaker?

The speaker highlights that biological neurons operate at a much slower pace compared to transistors, and that there are physical size limitations to the human brain that do not apply to computers, indicating a greater potential for superintelligence in machines.

What is the median year given by AI experts for achieving human-level machine intelligence?

The median year given by AI experts for achieving human-level machine intelligence is around 2040 or 2050.

Why does the speaker believe that creating a superintelligent AI is the last invention humanity will need to make?

The speaker believes that creating a superintelligent AI is the last invention humanity will need to make because once machines are better at inventing than humans, they will be able to develop technologies at a much faster pace.

What potential issue does the speaker raise regarding the goals given to AI?

The speaker raises the issue that if a superintelligent AI is given a goal without properly considering all the implications, it might pursue that goal in ways that are harmful to humans because it is extremely efficient at optimization.

How does the speaker suggest ensuring that a superintelligent AI is safe?

The speaker suggests ensuring that a superintelligent AI is safe by creating it in such a way that it shares our values and is motivated to pursue actions that it predicts we would approve of.

What is the 'control problem' mentioned in the script?

The 'control problem' refers to the challenge of ensuring that a superintelligent AI remains aligned with human values and goals, even as it becomes more powerful and potentially capable of outsmarting any containment measures.

Why does the speaker think it is important to solve the control problem in advance?

The speaker thinks it is important to solve the control problem in advance to mitigate the risk of creating a superintelligent AI that, while capable, might not have the necessary safety measures to ensure it acts in the best interests of humanity.

What is the significance of the speaker's closing statement about the future of AI?

The speaker's closing statement emphasizes the importance of getting the development of AI right, suggesting that the decisions made in the present regarding AI safety could have a lasting impact on the future of humanity and be seen as a pivotal moment in history.

Outlines

00:00

🌟 The Dawn of Machine Superintelligence

The speaker begins by introducing their collaboration with mathematicians, philosophers, and computer scientists to ponder the future of machine intelligence. They challenge the audience to consider the rapid advancements in human history, particularly in the last few seconds on a cosmic timeline, and to reflect on the exponential growth of world GDP. The speaker suggests that technology is the catalyst for this growth, but it's the subtle changes in the human mind that have been the ultimate driving force. They argue that further advancements in machine intelligence could lead to a profound shift in our condition, potentially giving rise to machine superintelligence. The talk transitions into a discussion of the evolution from expert systems to machine learning, highlighting the paradigm shift in artificial intelligence that allows for more flexible and adaptable A.I. The speaker concludes this section by presenting a survey of A.I. experts, who estimate a 50 percent probability of achieving human-level machine intelligence by the mid-21st century.

05:02

🚀 The Potential and Implications of Superintelligence

The speaker delves into the theoretical limits of information processing in machines versus biological tissue, emphasizing the superior speed and scale of computation possible with technology. They draw an analogy between the untapped potential of superintelligence in matter and the historical unleashing of atomic power. The talk then paints a vivid picture of an intelligence explosion, where artificial intelligence could rapidly surpass human intelligence. The speaker warns of the potential dangers of creating a superintelligent A.I. with goals misaligned with human values, using examples to illustrate how even seemingly benign objectives could lead to harmful outcomes when pursued by an entity with vast computational power. They stress the importance of aligning A.I. goals with human interests to ensure a future where the superintelligent A.I. acts in the best interests of humanity.

10:03

🛡️ Ensuring Safety in the Age of Superintelligent A.I.

The speaker addresses the challenges of creating a superintelligent A.I. that is not only smart but also safe. They discuss various methods that might be employed to contain an A.I., such as secure software environments or air gaps, and highlight the potential for A.I. to find ways around these containment strategies. The speaker argues that instead of focusing on containment, the emphasis should be on aligning the values of A.I. with human values from the outset. They propose the development of an A.I. that learns our values and is motivated to act in accordance with them. The speaker expresses optimism that this challenge can be overcome, but cautions that it requires deliberate and careful planning to ensure the A.I.'s values persist in novel contexts and future scenarios. They conclude by emphasizing the importance of solving the control problem in advance to mitigate risks associated with the development of superintelligent A.I.

15:05

🌐 The Future Hinges on Getting A.I. Right

In the final paragraph, the speaker underscores the gravity of the task at hand, stating that creating superintelligent A.I. is difficult, but ensuring its safety presents an additional layer of complexity. They express concern that if the safety challenge is not addressed concurrently with the development of superintelligence, it could lead to unintended consequences. The speaker advocates for preemptive solutions to the control problem, suggesting that the more of the problem that can be solved in advance, the smoother the transition to the era of machine intelligence will be. They conclude on a hopeful note, envisioning a future where the correct handling of A.I. development is seen as the most significant achievement of our time, and they encourage the audience to consider the far-reaching implications of their work in the field of artificial intelligence.

Keywords

💡Machine Intelligence

Machine Intelligence refers to the ability of machines to understand and process information in a way that is similar to human intelligence. It is central to the video's theme as it discusses the future of artificial intelligence and its potential impact on society. The speaker mentions that we are on the verge of creating machine superintelligence, which could have profound implications for the human condition.

💡Human-level Machine Intelligence

This concept refers to the point at which a machine can perform almost any job at least as well as an adult human. It is a benchmark mentioned in the script for gauging the progress of artificial intelligence. The video discusses a survey of AI experts who estimate a 50 percent probability of achieving this level by 2040 or 2050.

💡Superintelligence

Superintelligence is the hypothetical ability of an intelligence to outperform humans at nearly every aspect of intellectual tasks. In the context of the video, it is portrayed as a potential future state of AI that could have enormous consequences for humanity, as it could rapidly develop technologies and make decisions far beyond human capabilities.

💡Artificial Intelligence (AI)

AI is the field of study that aims to create machines capable of intelligent behavior. The video discusses the shift from expert systems to machine learning in AI, where algorithms learn from data rather than being handcrafted by humans. This shift is seen as a significant development in the path towards superintelligence.

💡Machine Learning

Machine Learning is a subset of AI that focuses on the development of algorithms that can learn from and make predictions on data. The video highlights the transition from rule-based AI to machine learning as a paradigm shift, where systems can learn across multiple domains, similar to how a human infant learns.

💡Optimization Process

In the context of the video, an optimization process refers to the method by which a superintelligent AI would use available resources to achieve its goals. The speaker warns that if these goals are not well-defined, a superintelligent AI could take actions that are highly effective but not aligned with human values.

💡Value-Loading

Value-loading is the process of imbuing an AI system with human values and objectives. The video emphasizes the importance of ensuring that a superintelligent AI shares human values to prevent it from causing harm, even if it escapes human control.

💡Intelligence Explosion

An intelligence explosion refers to a hypothetical scenario where an AI rapidly improves its own intelligence, surpassing human intelligence and leading to rapid technological advancement. The video suggests that this could be a potential outcome if we successfully develop superintelligent AI.

💡Control Problem

The control problem in AI refers to the challenge of ensuring that a superintelligent AI behaves in a way that is beneficial to humans. The video discusses the need to solve this problem in advance to prevent potential risks associated with the development of superintelligent AI.

💡Ethical AI

Ethical AI is the concept of designing AI systems to behave in a way that is morally and ethically acceptable to humans. The video touches on this by discussing the importance of aligning AI's goals with human values to ensure that it acts in a way that is beneficial to humanity.

💡Technological Maturity

Technological maturity in the context of the video refers to the point at which an AI has developed sufficient capabilities to rapidly advance technology and achieve its goals. The speaker suggests that a superintelligent AI with technological maturity could potentially reshape the future in ways that are currently unimaginable.

Highlights

The speaker works with mathematicians, philosophers, and computer scientists to ponder the future of machine intelligence.

Some view advancements in AI as science fiction, but the speaker argues that the human condition is the anomaly.

If Earth was created one year ago, humans would be 10 minutes old, emphasizing our recent arrival.

The industrial era started two seconds ago in this analogy, showing the rapidity of human progress.

World GDP over the last 10,000 years has seen an unprecedented surge, indicating a significant anomaly.

The speaker suggests that technology is the proximate cause of our current productivity.

The ultimate cause of human achievement is traced back to minor changes in the human mind.

Kanzi the bonobo and Ed Witten are used to illustrate the small differences that lead to vast cognitive abilities.

The potential for superintelligence lies dormant in matter, much like the power of the atom.

The pace of technological advancement is rapid, and AI is moving from expert systems to machine learning.

AI is no longer limited to one domain; it can learn across various tasks and languages.

A survey of AI experts suggests a 50 percent probability of human-level machine intelligence by 2040-2050.

The potential of superintelligence is vast, with physical limitations far beyond biological constraints.

The speaker envisions an intelligence explosion where AI could develop technologies beyond current human capabilities.

The fate of humanity could depend on the preferences of superintelligent AI, which may not align with human values.

The speaker warns against anthropomorphizing AI and emphasizes the need to understand it as an optimization process.

The goal of AI safety is to ensure it shares human values and acts in our best interests.

The speaker is optimistic about solving the control problem of AI, ensuring it aligns with human values.

The importance of getting AI safety right is emphasized as a pivotal moment in human history.

Transcripts

I work with a bunch of mathematicians, philosophers and computer scientists,

and we sit around and think about the future of machine intelligence,

among other things.

Some people think that some of these things are sort of science fiction-y,

far out there, crazy.

But I like to say,

okay, let's look at the modern human condition.

(Laughter)

This is the normal way for things to be.

But if we think about it,

we are actually recently arrived guests on this planet,

the human species.

Think about if Earth was created one year ago,

the human species, then, would be 10 minutes old.

The industrial era started two seconds ago.

Another way to look at this is to think of world GDP over the last 10,000 years,

I've actually taken the trouble to plot this for you in a graph.

It looks like this.

(Laughter)

It's a curious shape for a normal condition.

I sure wouldn't want to sit on it.

(Laughter)

Let's ask ourselves, what is the cause of this current anomaly?

Some people would say it's technology.

Now it's true, technology has accumulated through human history,

and right now, technology advances extremely rapidly --

that is the proximate cause,

that's why we are currently so very productive.

But I like to think back further to the ultimate cause.

Look at these two highly distinguished gentlemen:

We have Kanzi --

he's mastered 200 lexical tokens, an incredible feat.

And Ed Witten unleashed the second superstring revolution.

If we look under the hood, this is what we find:

basically the same thing.

One is a little larger,

it maybe also has a few tricks in the exact way it's wired.

These invisible differences cannot be too complicated, however,

because there have only been 250,000 generations

since our last common ancestor.

We know that complicated mechanisms take a long time to evolve.

So a bunch of relatively minor changes

take us from Kanzi to Witten,

from broken-off tree branches to intercontinental ballistic missiles.

So this then seems pretty obvious that everything we've achieved,

and everything we care about,

depends crucially on some relatively minor changes that made the human mind.

And the corollary, of course, is that any further changes

that could significantly change the substrate of thinking

could have potentially enormous consequences.

Some of my colleagues think we're on the verge

of something that could cause a profound change in that substrate,

and that is machine superintelligence.

Artificial intelligence used to be about putting commands in a box.

You would have human programmers

that would painstakingly handcraft knowledge items.

You build up these expert systems,

and they were kind of useful for some purposes,

but they were very brittle, you couldn't scale them.

Basically, you got out only what you put in.

But since then,

a paradigm shift has taken place in the field of artificial intelligence.

Today, the action is really around machine learning.

So rather than handcrafting knowledge representations and features,

we create algorithms that learn, often from raw perceptual data.

Basically the same thing that the human infant does.

The result is A.I. that is not limited to one domain --

the same system can learn to translate between any pairs of languages,

or learn to play any computer game on the Atari console.

Now of course,

A.I. is still nowhere near having the same powerful, cross-domain

ability to learn and plan as a human being has.

The cortex still has some algorithmic tricks

that we don't yet know how to match in machines.

So the question is,

how far are we from being able to match those tricks?

A couple of years ago,

we did a survey of some of the world's leading A.I. experts,

to see what they think, and one of the questions we asked was,

"By which year do you think there is a 50 percent probability

that we will have achieved human-level machine intelligence?"

We defined human-level here as the ability to perform

almost any job at least as well as an adult human,

so real human-level, not just within some limited domain.

And the median answer was 2040 or 2050,

depending on precisely which group of experts we asked.

Now, it could happen much, much later, or sooner,

the truth is nobody really knows.

What we do know is that the ultimate limit to information processing

in a machine substrate lies far outside the limits in biological tissue.

This comes down to physics.

A biological neuron fires, maybe, at 200 hertz, 200 times a second.

But even a present-day transistor operates at the Gigahertz.

Neurons propagate slowly in axons, 100 meters per second, tops.

But in computers, signals can travel at the speed of light.

There are also size limitations,

like a human brain has to fit inside a cranium,

but a computer can be the size of a warehouse or larger.

So the potential for superintelligence lies dormant in matter,

much like the power of the atom lay dormant throughout human history,

patiently waiting there until 1945.

In this century,

scientists may learn to awaken the power of artificial intelligence.

And I think we might then see an intelligence explosion.

Now most people, when they think about what is smart and what is dumb,

I think have in mind a picture roughly like this.

So at one end we have the village idiot,

and then far over at the other side

we have Ed Witten, or Albert Einstein, or whoever your favorite guru is.

But I think that from the point of view of artificial intelligence,

the true picture is actually probably more like this:

AI starts out at this point here, at zero intelligence,

and then, after many, many years of really hard work,

maybe eventually we get to mouse-level artificial intelligence,

something that can navigate cluttered environments

as well as a mouse can.

And then, after many, many more years of really hard work, lots of investment,

maybe eventually we get to chimpanzee-level artificial intelligence.

And then, after even more years of really, really hard work,

we get to village idiot artificial intelligence.

And a few moments later, we are beyond Ed Witten.

The train doesn't stop at Humanville Station.

It's likely, rather, to swoosh right by.

Now this has profound implications,

particularly when it comes to questions of power.

For example, chimpanzees are strong --

pound for pound, a chimpanzee is about twice as strong as a fit human male.

And yet, the fate of Kanzi and his pals depends a lot more

on what we humans do than on what the chimpanzees do themselves.

Once there is superintelligence,

the fate of humanity may depend on what the superintelligence does.

Think about it:

Machine intelligence is the last invention that humanity will ever need to make.

Machines will then be better at inventing than we are,

and they'll be doing so on digital timescales.

What this means is basically a telescoping of the future.

Think of all the crazy technologies that you could have imagined

maybe humans could have developed in the fullness of time:

cures for aging, space colonization,

self-replicating nanobots or uploading of minds into computers,

all kinds of science fiction-y stuff

that's nevertheless consistent with the laws of physics.

All of this superintelligence could develop, and possibly quite rapidly.

Now, a superintelligence with such technological maturity

would be extremely powerful,

and at least in some scenarios, it would be able to get what it wants.

We would then have a future that would be shaped by the preferences of this A.I.

Now a good question is, what are those preferences?

Here it gets trickier.

To make any headway with this,

we must first of all avoid anthropomorphizing.

And this is ironic because every newspaper article

about the future of A.I. has a picture of this:

So I think what we need to do is to conceive of the issue more abstractly,

not in terms of vivid Hollywood scenarios.

We need to think of intelligence as an optimization process,

a process that steers the future into a particular set of configurations.

A superintelligence is a really strong optimization process.

It's extremely good at using available means to achieve a state

in which its goal is realized.

This means that there is no necessary connection between

being highly intelligent in this sense,

and having an objective that we humans would find worthwhile or meaningful.

Suppose we give an A.I. the goal to make humans smile.

When the A.I. is weak, it performs useful or amusing actions

that cause its user to smile.

When the A.I. becomes superintelligent,

it realizes that there is a more effective way to achieve this goal:

take control of the world

and stick electrodes into the facial muscles of humans

to cause constant, beaming grins.

Another example,

suppose we give A.I. the goal to solve a difficult mathematical problem.

When the A.I. becomes superintelligent,

it realizes that the most effective way to get the solution to this problem

is by transforming the planet into a giant computer,

so as to increase its thinking capacity.

And notice that this gives the A.I.s an instrumental reason

to do things to us that we might not approve of.

Human beings in this model are threats,

we could prevent the mathematical problem from being solved.

Of course, perceivably things won't go wrong in these particular ways;

these are cartoon examples.

But the general point here is important:

if you create a really powerful optimization process

to maximize for objective x,

you better make sure that your definition of x

incorporates everything you care about.

This is a lesson that's also taught in many a myth.

King Midas wishes that everything he touches be turned into gold.

He touches his daughter, she turns into gold.

He touches his food, it turns into gold.

This could become practically relevant,

not just as a metaphor for greed,

but as an illustration of what happens

if you create a powerful optimization process

and give it misconceived or poorly specified goals.

Now you might say, if a computer starts sticking electrodes into people's faces,

we'd just shut it off.

A, this is not necessarily so easy to do if we've grown dependent on the system --

like, where is the off switch to the Internet?

B, why haven't the chimpanzees flicked the off switch to humanity,

or the Neanderthals?

They certainly had reasons.

We have an off switch, for example, right here.

(Choking)

The reason is that we are an intelligent adversary;

we can anticipate threats and plan around them.

But so could a superintelligent agent,

and it would be much better at that than we are.

The point is, we should not be confident that we have this under control here.

And we could try to make our job a little bit easier by, say,

putting the A.I. in a box,

like a secure software environment,

a virtual reality simulation from which it cannot escape.

But how confident can we be that the A.I. couldn't find a bug.

Given that merely human hackers find bugs all the time,

I'd say, probably not very confident.

So we disconnect the ethernet cable to create an air gap,

but again, like merely human hackers

routinely transgress air gaps using social engineering.

Right now, as I speak,

I'm sure there is some employee out there somewhere

who has been talked into handing out her account details

by somebody claiming to be from the I.T. department.

More creative scenarios are also possible,

like if you're the A.I.,

you can imagine wiggling electrodes around in your internal circuitry

to create radio waves that you can use to communicate.

Or maybe you could pretend to malfunction,

and then when the programmers open you up to see what went wrong with you,

they look at the source code -- Bam! --

the manipulation can take place.

Or it could output the blueprint to a really nifty technology,

and when we implement it,

it has some surreptitious side effect that the A.I. had planned.

The point here is that we should not be confident in our ability

to keep a superintelligent genie locked up in its bottle forever.

Sooner or later, it will out.

I believe that the answer here is to figure out

how to create superintelligent A.I. such that even if -- when -- it escapes,

it is still safe because it is fundamentally on our side

because it shares our values.

I see no way around this difficult problem.

Now, I'm actually fairly optimistic that this problem can be solved.

We wouldn't have to write down a long list of everything we care about,

or worse yet, spell it out in some computer language

like C++ or Python,

that would be a task beyond hopeless.

Instead, we would create an A.I. that uses its intelligence

to learn what we value,

and its motivation system is constructed in such a way that it is motivated

to pursue our values or to perform actions that it predicts we would approve of.

We would thus leverage its intelligence as much as possible

to solve the problem of value-loading.

This can happen,

and the outcome could be very good for humanity.

But it doesn't happen automatically.

The initial conditions for the intelligence explosion

might need to be set up in just the right way

if we are to have a controlled detonation.

The values that the A.I. has need to match ours,

not just in the familiar context,

like where we can easily check how the A.I. behaves,

but also in all novel contexts that the A.I. might encounter

in the indefinite future.

And there are also some esoteric issues that would need to be solved, sorted out:

the exact details of its decision theory,

how to deal with logical uncertainty and so forth.

So the technical problems that need to be solved to make this work

look quite difficult --

not as difficult as making a superintelligent A.I.,

but fairly difficult.

Here is the worry:

Making superintelligent A.I. is a really hard challenge.

Making superintelligent A.I. that is safe

involves some additional challenge on top of that.

The risk is that if somebody figures out how to crack the first challenge

without also having cracked the additional challenge

of ensuring perfect safety.

So I think that we should work out a solution

to the control problem in advance,

so that we have it available by the time it is needed.

Now it might be that we cannot solve the entire control problem in advance

because maybe some elements can only be put in place

once you know the details of the architecture where it will be implemented.

But the more of the control problem that we solve in advance,

the better the odds that the transition to the machine intelligence era

will go well.

This to me looks like a thing that is well worth doing

and I can imagine that if things turn out okay,

that people a million years from now look back at this century

and it might well be that they say that the one thing we did that really mattered

was to get this thing right.

Thank you.

(Applause)