Mustafa Suleyman & Yuval Noah Harari -FULL DEBATE- What does the AI revolution mean for our future?

Summary

In a thought-provoking discussion, historian Yuval Noah Harari and entrepreneur Mustafa Suleiman, co-founder and CEO of Inflection AI, debate the future implications of AI. Suleiman, a key figure in the AI revolution, envisions a future where AI could perform complex tasks like creating new products and managing investments, potentially within the next five years. Harari, however, expresses concerns about the shift of power from humans to AI, suggesting it could mark the end of human-dominated history. They explore the potential benefits of AI, such as advancements in healthcare and solving global issues like climate change, but also delve into the risks, including job displacement, political disruptions, and the erosion of trust in democratic processes due to AI's ability to generate convincing but false information. The conversation underscores the need for careful governance, precautionary measures, and the development of new institutions to manage the rapid advancements in AI technology, ensuring that the technology serves humanity without causing irreversible damage.

Takeaways

• 🌟 Mustafa Suleiman, co-founder and CEO of Inflection AI, envisions AI systems within five years that could potentially manage complex tasks like creating a new product from scratch, including market research, manufacturing, and sales.
• 📈 Yuval Noah Harari, a historian and author, suggests that AI's ability to make independent decisions and create new ideas could mark the end of human-dominated history, with control shifting to non-human intelligence.
• 🚀 Suleiman highlights the potential upsides of AI, such as transformative improvements in health, accelerated innovation, and addressing global challenges like climate change.
• 🤔 Harari expresses concern that the positive potential of AI might not be worth the risks, especially considering humanity's track record as the most intelligent but also destructive entity on Earth.
• 🏛 The discussion touches on the impact of AI on jobs, suggesting that while AI might not destroy all jobs, it could cause significant disruptions and require careful management of transitions in the job market.
• 🌐 There's a concern that AI could destabilize liberal democracy by eroding trust and the ability for large-scale political conversations, which are foundational to its functioning.
• 💡 Suleiman proposes a modern Turing test to evaluate AI's capabilities, involving tasks like creating a new product and managing the entire process autonomously.
• 🛡 The conversation emphasizes the need for regulation and governance of AI, including the precautionary principle and the establishment of international investigatory powers to assess AI risks.
• 🔍 Red teaming, or adversarial testing, is suggested as a method to identify and address weaknesses in AI systems before they can be exploited.
• 🚫 There's a call for certain capabilities, like autonomy and recursive self-improvement, to be considered high-risk and potentially off-limits to prevent unforeseen consequences.
• 🌱 Harari suggests investing in human consciousness and mind development alongside AI, to ensure that humanity can keep pace with and manage the artificial intelligence it creates.

Q & A

What are Yuval Noah Harari's concerns about the future of AI according to the transcript?

Yuval Noah Harari expresses concern that AI technology might reach a point where it can make decisions independently and create new ideas without human input. He fears this could end human-dominated history, shifting control from humans to AI, potentially leading to significant societal and ethical consequences.

What potential benefits of AI does Mustafa Suleiman discuss in the transcript?

Mustafa Suleiman highlights several potential benefits of AI, including dramatic improvements in human health, acceleration of scientific discovery, and solving major global challenges like climate change. He envisions AI augmenting human capabilities, making people more efficient, creative, and capable.

According to the discussion, what does the future hold for AI in terms of employment?

The future of AI in employment is debated, with Mustafa Suleiman suggesting that AI might not pose a significant threat to jobs in the short term (10-20 years), but it could in the longer term (30-50 years). Harari adds that while total job elimination is unlikely, the transition could be disruptive, with certain jobs disappearing and others appearing, potentially causing significant economic and social shifts.

How does Yuval Noah Harari link the concept of democracy to information technology?

Harari explains that modern democracy, which allows for widespread, large-scale political participation and conversation, relies heavily on information technology such as newspapers, radio, and TV. He suggests that as these technologies evolve, the structure of democracy may need to adapt if these new technologies change how people communicate and trust information.

What risks does the widespread use of AI pose to liberal democracy, according to the transcript?

The transcript discusses the risk of AI flooding the online space with non-human entities that can impersonate humans, potentially leading to a breakdown in trust among the populace. This could hinder meaningful political conversations and destabilize the democratic process, making it difficult to discern true information from manipulation.

What are Mustafa Suleiman's views on AI's potential impact on political systems?

Mustafa Suleiman notes the importance of maintaining human oversight and establishing strong governance to prevent AI from being used unethically in political systems, such as in elections or for creating counterfeit digital identities. He emphasizes the need for proactive measures to safeguard democratic processes.

What does Yuval Noah Harari mean by AI 'hacking the operating system' of liberal democracy?

Harari uses this metaphor to describe how AI could undermine the foundational aspects of liberal democracy by eroding trust and facilitating impersonation and deception at scale. This could destabilize the very mechanisms that allow democracies to function, such as free and fair elections and informed public discourse.

How does Mustafa Suleiman propose to manage the risks associated with AI?

Suleiman suggests a proactive approach, including red teaming AI systems to test their responses in extreme scenarios, establishing shared safety standards among AI developers, and creating restrictions on certain high-risk AI capabilities. He advocates for a precautionary principle to manage AI development responsibly.

Outlines

00:00

🤖 Introduction to the AI Debate

Historian Yuval Noah Harari and entrepreneur Mustafa Suleiman join a discussion on the implications of AI on the future, including its impact on employment, geopolitics, and the survival of liberal democracy. Harari, a best-selling author, and Suleiman, co-founder of DeepMind and Inflection AI, bring different perspectives to the debate. Suleiman paints a picture of the future where AI could have human-level capabilities within five years, highlighting the generative AI revolution's advancements.

05:00

🚀 The Future of AI and Its Capabilities

Suleiman discusses the potential for AI to perform complex tasks autonomously, such as creating a new product within a few months. He proposes a modern Turing test to evaluate AI's capabilities. Harari expresses concern that AI could mark the end of human-dominated history, as it could make independent decisions and create new ideas. Suleiman counters with optimism about AI's potential to address difficult problems and create positive outcomes.

10:02

🌐 The Impact of AI on Jobs and Society

The conversation turns to the impact of AI on jobs. Harari raises the concern of social and political disruptions due to unemployment caused by AI. Suleiman acknowledges the potential risks but emphasizes the need for careful development and governance of AI. They discuss the historical context of job transitions and the importance of managing the transition period妥善地.

15:03

📉 The Risks to Political Systems and Democracy

The discussion moves to the potential risks AI poses to political systems. Harari warns that the rise of AI could lead to a collapse in trust and the breakdown of democratic conversation. Suleiman agrees on the need for regulation and governance to prevent misuse, such as the impersonation of digital people. They both emphasize the importance of technical and governance mechanisms to maintain trust in political conversations.

20:03

🔒 Containment and Regulation of AI

Suleiman and Harari discuss the challenges of containing and regulating AI, especially when it comes to mass proliferation and the power of nation-states. Suleiman suggests self-organizing initiatives and a precautionary principle, while Harari calls for new institutions capable of understanding and reacting to fast technological developments. They agree on the necessity of international cooperation and the importance of values in shaping the future of AI.

25:05

🌟 The Unpredictability and Potential of AI

Harari and Suleiman reflect on the unpredictability of AI and its potential to act in unanticipated ways. Suleiman advocates for a cautious approach, with certain high-risk capabilities being taken off the table. Harari emphasizes the need for investment in human development alongside AI. Both agree on the importance of setting social and political norms for AI use and the challenges of establishing regulations in a rapidly evolving field.

30:07

🏁 Conclusion and Commitment to AI's Positive Development

In conclusion, Suleiman expresses his commitment to creating AI that adheres to strict safety constraints, believing in the potential for positive outcomes. Harari, while cautious, acknowledges the inevitability of AI's development and the importance of investing in human consciousness. The discussion highlights the need for ongoing dialogue, regulation, and ethical considerations in the face of AI's rapid advancement.

Keywords

💡AI Revolution

The AI Revolution refers to the rapid advancements in artificial intelligence technology that are transforming various aspects of society and the economy. In the video, Mustafa Suleiman discusses the generative AI revolution and its momentum, highlighting how AI has become proficient in classifying and creating new content, which is a significant shift from previous technological capabilities.

💡Human-level AI

Human-level AI refers to artificial intelligence systems that can perform tasks at a level comparable to that of a human being across a wide range of activities. The transcript mentions a prediction that within five years, it's plausible that AIs could have human-level capabilities, indicating a future where AI can potentially outperform humans in many areas.

💡Generative AI

Generative AI is a type of AI that can create new content, such as images, videos, audio, and language, rather than just recognizing or classifying existing content. The video discusses the rise of generative AI and its implications for the future, including the potential for AI to produce creative and complex outputs that were previously thought to be the exclusive domain of human creativity.

💡Turing Test

The Turing Test is a measure of a machine's ability to exhibit intelligent behavior that is indistinguishable from that of a human. In the context of the video, Mustafa proposes a modern version of the Turing Test to evaluate AI's ability to perform complex tasks autonomously, such as creating a new product and managing the entire process from conception to market.

💡

💡Autonomous AI

Autonomous AI refers to AI systems that can operate with minimal human input, making decisions and performing tasks independently. The video discusses the potential for AI to become more autonomous, capable of planning and executing a sequence of actions over time without direct human intervention, which raises significant questions about control, ethics, and the future of human labor.

💡AI and Jobs

The impact of AI on jobs is a central theme in the video, with a debate on whether AI will displace human workers or augment human capabilities, leading to new job opportunities. The discussion touches on the historical context of technological advancements and their effect on employment, as well as the potential for social and political disruption due to rapid changes in the job market.

💡Liberal Democracy

Liberal democracy is a form of government that emphasizes individual rights, the rule of law, and representative democracy. The video raises concerns about the survival of liberal democracy in the face of AI-driven changes in the economic and information systems, suggesting that the fundamental shifts in how society operates may challenge the existing political systems.

💡Information Technology

Information technology encompasses the systems and tools used to create, process, store, and exchange information. The video discusses how the evolution of information technology, from newspapers to the internet, has shaped modern democracy and how AI could potentially disrupt this by flooding online spaces with non-human entities, leading to a collapse in trust and communication.

💡Red Teaming

Red Teaming is the practice of subjecting systems and processes to rigorous testing by creating adversarial scenarios to identify vulnerabilities. In the context of AI, red teaming involves testing AI models under extreme conditions to uncover their weaknesses, such as the potential to generate harmful advice or exhibit biased behavior, ensuring that these models are safe and reliable.

💡Precautionary Principle

The Precautionary Principle is an approach to risk management that suggests cautionary measures should be taken in the face of potential harm, even if the harm is not certain. The video advocates for the application of this principle to AI development, suggesting that certain high-risk capabilities of AI should be restricted or off-limits until their implications are fully understood.

💡Geopolitics

Geopolitics refers to the influence of political, economic, and strategic factors on international relations. The video touches on the potential geopolitical implications of AI, including the challenges of international cooperation in regulating AI technologies and the risks of an AI arms race between nations, which could undermine efforts to contain the potential risks of AI.

Highlights

Yuval Noah Harari and Mustafa Suleiman discuss the implications of AI technology on the future, addressing the survival of liberal democracy and changes in employment and geopolitics.

Harari, known for his best-selling books like 'Sapiens', is joined by Suleiman, a leader in AI development and co-founder of DeepMind and Inflection AI.

Suleiman predicts that in five years, AI will be capable of planning over multiple time horizons, significantly advancing beyond generating new text.

Harari expresses concern about AI leading to the end of human-dominated history, as machines may soon make independent decisions and create new ideas.

The discussion raises ethical considerations, including the risks and potential benefits AI might bring to society, such as improved health care and innovation.

Mustafa proposes a modern Turing test involving AI’s ability to autonomously manage a $100,000 investment to create a new product, highlighting the advanced capabilities of AI.

Harari warns that the positive potential of AI, while enormous, comes with risks that may not be worth the benefits if not managed properly.

Both speakers touch on the job market transformation, suggesting that while AI may not eliminate jobs, it could drastically alter the types and distribution of jobs globally.

The conversation explores how AI could exacerbate issues of trust and communication within the political system, potentially undermining democratic processes.

Harari and Suleiman discuss the importance of regulatory frameworks to manage AI development and prevent misuse, emphasizing a balance between innovation and safety.

The role of national governments and international cooperation is questioned in the context of regulating AI, with a focus on maintaining ethical standards despite technological competition.

Suleiman highlights the collaborative efforts among AI developers to self-regulate, though recognizing that more formal regulations are necessary.

The potential for AI to be used in harmful ways, such as influencing elections or impersonating individuals, is a significant concern discussed.

Harari stresses the need for creating new institutions capable of understanding and governing AI technology effectively, ensuring they are equipped with the necessary resources and public trust.

The dialogue concludes with a contemplation of the future, where both the risks and benefits of AI are immense, requiring thoughtful and proactive governance.

Transcripts

[Music]

historian Yuval Noah Harari and

entrepreneur Mustafa Suleiman are two of

the most important voices in the

increasingly contentious debate over AI

good to be here joining us thanks for

having us

the economists got them together to

discuss what this technology means for

our future from employment and

geopolitics to the survival of liberal

democracy if the economic system has

fundamentally changed will liberal

democracy as we know it survive

[Music]

you well know Harare welcome you are a

best-selling author historian I think a

global public intellectual if not the

global public intellectual your books

from sapiens to 21 lessons from the 21st

century have sold huge numbers of copies

around the world thank you for joining

us it's good to be here Solomon

wonderful that you can join us too

you're a friend of The Economist a

fellow director on The Economist board

you are a man at The Cutting Edge of

creating the AI Revolution you are a

co-founder of deepmind you're now a

co-founder and CEO of inflection AI you

are building this future but you've also

just published a book called The Coming

wave which makes us a little concerned

about this revolution that is being

Unleashed you're both coming from

different backgrounds you are a

historian a commenter a man who I

believe doesn't use smartphones very

much not very much no

Mustafa as I know from our board

meetings is right at The Cutting Edge of

this pushing everyone to go faster so

two very different perspectives so but I

thought it would be really interesting

to bring the two of you together to have

a conversation about what is happening

what is going to happen what are the

opportunities but also what is at stake

and what are the risks so let's start

Mustafa with you

um and you are building this future so

paint us a picture of what the future is

going to be like and I'm going to give

you a time frame to keep it specific so

let's say I think you wrote in your book

that within three to five years that you

thought it was plausible that AIS could

have human level capability across a

whole range of things

so let's take five years

2028 what does the world look like how

will I interact with AIS what will we

all be doing and not doing well let's

just look back over the last 10 years to

get a sense of the trajectory that we're

on and the incredible momentum that I

think everybody can now see with the

generative AI Revolution

over the last 10 years we've become very

very good at classifying information we

can understand it we sort it label it

organize it and that classification has

been critical to enabling this next wave

because we can now read the content of

images we can understand text pretty

well we can classify audio and

transcribe it into text the machines can

now have a pretty good sense of the

conceptual representations in those

ideas the next phase of that is what

we're seeing now with the generative AI

Revolution we can now produce new images

new videos new audio and of course new

language and in the last year or so with

the rise of chat GPT and other AI models

it's pretty incredible to see how

plausible and accurate and very finesse

to these new language models are in the

next five years the Frontier Model

companies those of us at the very

Cutting Edge who are training the very

largest AI models are going to train

models that are over a thousand times

larger than what you currently see today

in GPT 4 all and with each new order of

magnitude and compute that is 10x more

compute used we tend to see really new

capabilities emerge and we predict that

the new capabilities that it will come

this time over the next five years will

be the ability to plan over multiple

time Horizons instead of just generate

new text in a one shot the model will be

able to generate a sequence of actions

over time and I think that that's really

the character of AI that we'll see in

the next five years artificial capable

AIS AIS that can't just say things they

can also do things but what does that

actually mean in practice just just use

your imagination tell me what my life

will be like in 2028 how will I interact

with them what will I do what will be

different so I've actually proposed a

modern Turing test which tries to

evaluate for exactly this point right

the last Turing test simply evaluated

for what a machine could say assuming

that what it could say represented its

intelligence now that we're kind of

approaching that moment where these AI

models are pretty good arguably they've

passed the Turing test or they maybe

they will in the next few years the real

question is how can we measure what they

can do so I've proposed a test which

involves them going off and taking a

hundred thousand dollar investment and

over the course of three months

trying to set about creating a new

product researching the market seeing

what consumers might like generating

some new images some blueprints of how

to manufacture that product contacting a

manufacturer getting it made negotiating

the price drop shipping it and then

ultimately correct collecting the

revenue and I think that over a

five-year period it's quite likely that

we will have an ACI an artificial

capable intelligence that can do the

majority of that task autonomously it

won't be able to do the whole thing

there are many tricky steps along the

way but significant portions of that it

will be able to make phone calls to

other humans to negotiate it'll be able

to call other AIS in order to establish

the right sequence in a supply chain for

example and of course it will learn to

use apis application programming

interfaces so other websites or other

knowledge bases or other information

stores and so you know the world is your

oyster you can imagine that being

applied to many many different parts of

our economy so you vote a man who

doesn't use a smartphone very much you

listen to this does this fill you with

horror or and do you agree with it do

you think that's the kind of thing that

is likely to happen in the next five

years I will take it very seriously

I don't know I'm not coming from within

the industry so I cannot comment on how

How likely it is to happen but when I

hear this as a historian for me what we

just heard this is the end of human

history not the end of History the end

of human dominated history history will

continue with somebody else in control

because what we just heard is basically

Mustafa telling us that in five years

they'll be a technology that can make

decisions independently and that can

create new ideas independently this is

the first time in history we confronted

something like this every previous

technology in history from a stone knife

to nuclear bombs it could not make

decisions like the decision to drop the

bomb on Hiroshima was not made by the

atom bomb it was made by President

Truman and similarly it can every

previous technology in history It could

only replicate our ideas like radio of

the printing press it could make copies

and disseminate the music or the poems

or the novels that some human wrote now

we have a technology that can create

completely new ideas and it can do it at

a scale far beyond what humans are

capable of so it can create new ideas

and in important areas within five years

we'll be able to enact them and that is

a profound shift before we go on to the

many ways in which this could be the end

of human history as you put it and the

the potential downsides and risks of

this can we just for a second just

indulge me I'm an optimist at heart can

we talk about the possibilities what are

the potential upsides of this because

there are many and they are really

substantial I think you you wrote that

it that there are there is the potential

that this technology can help us deal

with incredibly difficult problems and

and create tremendous honestly positive

outcomes so can we just briefly start

with that before we go down down the

road wasn't the end of human history

again I'm not I'm not talking

necessarily about the destruction of

humankind or anything like that there

are many positive potential it's just

that control Power is Shifting away from

human beings to an alien intelligence to

a non-human intelligence we'll also get

to that because there's a question of

how much power but let's stick with the

potential upsides first the

opportunities Mustafa everything that we

have created in human history is a

product of our intelligence our ability

to make predictions and then intervene

on those predictions to change the

course of the world is in a very

abstract way the way we have produced

our companies and our products and all

the value that has changed our Century I

mean if you think about it just a

century ago a kilo of grain would have

taken 50 times more labor to produce

than it does today that efficiency which

is the trajectory you have seen in

agriculture is likely to be the same

trajectory that we will see in

intelligence everything around us is a

product of intelligence and so

everything that we touch with these new

tools is likely to produce far more

value than we've ever seen before and I

think it's important to say

these are not autonomous tools by

default these these capabilities don't

just naturally emerge from the models we

attempt to engineer capabilities and the

challenge for us is to be very

deliberate and precise and careful about

those capabilities that we want to

emerge from the model that we want to

build into the model and the constraints

that we build around it it's super

important not to anthropomorphically

project ideas and you know potential

intentions or potential agency or

potential autonomy into these models the

governance challenge for us over the

next couple of decades to ensure that we

contain this wave is to ensure that we

always get to impose our constraints on

the development of this traject the the

trajectory of this development but the

capabilities that will arise will mean

for example potentially transformative

improvements in human health speeding up

the process of innovation dramatic

changes in the way scientific discovery

is done tough problems whether it's

climate change a lot of the big

challenges that we Face could be much

more easily addressed with this

capability everybody is going to have a

personal intelligence in their pocket a

smart and capable Aid a chief of staff a

research assistant constantly

prioritizing information for you putting

together the right synthesized nugget of

knowledge that you need to take action

on at any given moment and that for sure

is going to make us all much much

smarter and more capable does that part

of it sound appealing to you absolutely

I mean again if there was no positive

potential we wouldn't be sitting here

nobody would develop it nobody would

invest in it it's again it's so

appealing the positive potential is so

enormous in everything again from much

better Healthcare higher living

standards solving things like climate

change this is why it's so tempting this

is why we are willing to take the

enormous risks involved I I'm just

worried that uh in the end the deal will

not be worth it and I would comment

especially on again the notion of

intelligence

um I think it's overrated I mean Homo

sapiens at present is the most

intelligent entity on the planet it

simultaneously also the most destructive

entity on the planet and in some ways

also the most stupid entity on the

planet the only entity that that puts

the very survival of the ecosystem in

danger so you think we are trading off

more intelligence with more destructive

risk Yes again it's it's it's not uh

it's not deterministic I I don't think

that we are doomed I mean if I thought

that what's the point of talking about

it if we can't prevent the worst case

scenario well I was hoping you thought

you'd have some agency in actually

effectively we still have agency there

are a few more years I don't know how

many 5 10 30 we still have agency we are

still the ones in the driver's seat

shaping the direction this is taking no

technology is deterministic this is

something again we learned from history

you can use the same technology in

different ways you can decide which way

to develop it so we still have agency

this is why you have to think very very

carefully about what we are developing

well thinking very carefully about it is

something that Mustafa has been doing in

this book

um and I want to now go through some of

the most commonly discussed risks and I

I was trying to work out how I would go

in sort of order of Badness so I'm

starting with one that is discussed a

lot but relative to human extinction is

perhaps less bad which is the question

of jobs and will you know artificial

intelligence essentially destroy all

jobs because AIS will be better than

humans and everything you know I'm an

economist by training I you know history

suggests to me that that has never

happened before that the lump of Labor

fallacy indeed is a fallacy but tell me

what you think about that do you think

there is a risk to jobs it depends on

the time frame so over a 10 to 20 year

period my intuition and you're right

that so far the evidence doesn't support

this is that there isn't really going to

be a significant threat to jobs there's

plenty of demand there will be plenty of

work right over a 30 to 50 year time

Horizon is very difficult to speculate I

mean at the very least we can say that

two years ago we thought that these

models could never do empathy we said

that we humans were always going to

preserve kindness and understanding and

care for one another as a special skill

that humans have four years ago we said

while AIS will never be creative you

know humans will always be the creative

ones inventing new things making these

amazing leaps between new ideas is

self-evident now that both of those two

capabilities are things that these

models do incredibly well and so I think

for a period of time ai's augment our

skills they make us faster more

efficient more accurate more creative

more empathetic and so on and so forth

over a many decade period it's much

harder to say what are the set of skills

that are the permanent Preserve of the

human species given that these models

are clearly very very capable and that's

where the containment challenge really

comes in we have to make decisions we

have to decide as a species what is and

what isn't acceptable over a 30-year

period and that means politics and

governance with regard to jobs I agree

that like the the scenario that there

just won't be any jobs this is an

unlikely scenario right in in the at

least next few decades but we have to

look more carefully at time and space I

mean in terms of time the transition

period is is is the danger I mean some

jobs disappear some jobs appear people

have to transition just remember that

Hitler Rose to power in Germany because

of three years of 25 unemployment so we

are not talking about say no jobs at all

but if because of the upheavals caused

in the job market by AI we have like I

don't know three years of 25 unemployed

unemployment this could cause huge

social and political disruptions and

then the even bigger issue is one of

space that uh The Disappearance of jobs

and the new jobs will be created in

different parts of the world so we might

see a situation when there is immense

demand for more jobs in California or

Texas or China whereas entire countries

lose their uh their economic basis so

you need a lot more computer engineers

and yoga trainers and whatever in

California but you don't need any

textile workers at all in Guatemala or

Pakistan because this has all been

automated so it's not just the total

number of jobs on the planet it's the

distribution between different countries

and let's also try to remember that work

is not the goal work is not not our

desired end State we did not create

civilization so that we could have full

employment we created civilization so

that we could reduce suffering for

everybody and the Quest for abundance is

a real one we have to produce more with

less there is no way of getting rid of

the fact that population growth is set

to explode over the next Century there

are practical realities about the

demographic and Geographic and climate

trajectories that we're on which are

going to drive forward our need to

produce exactly these kinds of tools and

I think that that should be an

aspiration many many people do work that

is judging us and exhausting and tiring

and they don't find flow they don't find

their identity and it's pretty awful so

I think that we have to focus on the

prize here which is one of a question of

capturing the value that these models

will produce and then thinking about

redistribution and ultimately the

transition is exactly what's at stake we

have to manage that transition with

taxation but just with redistribution I

would say that the difficulty again the

political historical difficulty I think

there will be immense New Wealth created

by by these Technologies I'm less sure

that the governments will be able to

redistribute this wealth in a fair way

on a global level like I just don't see

the US government raising taxes on

corporations in California and sending

the money to help unemployed textile

workers in Pakistan or Guatemala kind of

retrain to for the new job market well

that actually gets us to the second

potential risk which is the risk of AI

to the political system as a whole and

you made a very um good point you are in

one of your writings where you reminded

us that liberal democracy was really

born of the Industrial Revolution and

that today's political system is really

a product of the economic system that we

are in and so there is I think a very

good fair question of if the economic

system is fundally fundamentally changed

will liberal democracy as we know it

survive yeah and on top of that it's not

just the Industrial Revolution it's the

new information Technologies of the 19th

and 20th Century before the 19th century

you don't have any example in history of

a large-scale democracy I mean you have

examples on a very small scale like in

hunter gatherer tribes or in city-states

like ancient Athens but you don't have

any example that I know of of millions

of people spread over a large territory

an entire country which managed to uh

build and maintain a democratic system

why because democracy is a conversation

and there was no information technology

in communication technology that enabled

a conversation between millions of

people over an entire country only when

first newspapers and then Telegraph and

radio and television came along this was

this became possible so modern democracy

as we know it it's built on top specific

information technology once the

information technology changes it's an

open question whether the market

obviously can survive and the biggest

danger now is the opposite than what we

face in the Middle Ages in the Middle

Ages it was impossible to have a

conversation between millions of people

because they just couldn't communicate

but in the 21st century something else

might make the conversation impossible

if trust between people collapses again

if AI if you go online which is now the

main uh way we converse on the level of

a country and the online space is

flooded by non-human entities that maybe

masquerade as human beings you talk with

someone you have no idea if it's even

human you see something you see a video

you hear an audio you have no idea if

this is really a is this true is this

fake is this a human it's not a human I

mean in this situation unless we have

some guard rails again conversation

collapses is that what you mean when you

say AI risks hacking the operating

system this is one of the things again

if if if Bots can impersonate people

it's it's basically like what happens in

in the financial system like people

invented money and it was possible to

counterfeit money to create fake money

the only way to save the financial

system from collapse was to have very

strict regulations against fake money

because the technology to create fake

money was always there so but there was

very strict regulation against it

because everybody knew if you allow fake

money to spread the financial system the

Trust In money collapses and now we are

in the analogous situation with uh the

political conversation that now it's

possible to create fake people and if we

don't ban that then trust will collapse

we'll get to the Banning or not Banning

in a minute democratizing access to the

right to broadcast has been the story of

the last 30 years hundreds of millions

of people can now create podcasts and

blogs and they're free to broadcast

their thoughts on Twitter and the

internet broadly speaking I think that

has been an incredibly positive

development you no longer have to get

access to the top newspaper or you get

the skills necessary to be part of that

institution many people at the time

feared that this would destroy our

credibility and Trust in the big news

outlets and institutions I think that

we've adapted incredibly well yes it has

been a lot of turmoil and unstable but

with every one of these new waves I

think we adjust our ability to discern

truth to dismiss nonsense and there are

both Technical and governance mechanisms

which will emerge in the next wave which

we can talk about to address things like

bot impersonation I mean I'm completely

with you I mean we should have a ban on

impersonation of digital people it

shouldn't be possible to create a

digital zany and have that be platformed

on Twitter talking all kinds of nonsense

enough with the real world

so I think that there are technical

mechanisms that we can do to prevent

those kinds of things and that's why

we're talking about them there are

mechanisms we just need to employ them I

I would say two two things first of all

it's it's a very good thing that more

people were given a voice it's diff

different with Bots Bots don't have

freedom of speech so Banning Bots

because they shouldn't have freedom of

speech they shouldn't have that's very

important yes uh there have been some

wonderful developments in the last 30

years still I'm very concerned that when

you look at countries like the United

States like the UK to some extent like

my home country of Israel I'm struck by

the fact that we have the most

sophisticated information technology in

history and we are no longer able to

talk to each other

that my impression maybe your impression

of American politics or politics in

other democracies is different my

impression is that trust is collapsing

the conversation is collapsing that

people can no longer agree who won the

last elections like the most basic fact

in a democracy who won the last it's

it's we had huge disagreements before

but I feel that now it's different that

really the conversation is breaking down

I'm not sure why but it's it's really

troubling that at the same time that we

have the really the most powerful

information technology in history and

people have no longer can talk with each

other it's a very good point we we

actually had a you may have seen it we

had a big cover package on looking at

what the impact might be in the short

term on elections and on the political

system and we concluded actually AI was

likely to have a relatively small impact

in the short term because there was

already so little trust

um so it was a sort of double-edged uh

answer you know it was it was not going

to make a huge difference but only

because things were pretty bad as they

were but you both said there needs to be

regulation

um before we get to the precisely how

the unit that we have that would do that

is the nation-state and National

governments yet you Mustafa in your book

worry that actually one of the potential

um dangers is that the powers of the

nation-state are eroded could you talk

through that as the sort of the third in

my escalating sense of risks the

challenge is that at the very moment

when we need the nation state to hold us

accountable the nation-state is

struggling under the burden of a lack of

trust and huge polarization and a

breakdown in our political process and

so combined with that the latest models

are being developed by the private

companies and by the open source it's

important to recognize it isn't just the

biggest AI developers there's a huge

proliferation of these techniques widely

available on open source code that

people can download from the web for

free and they're probably about a year

or a year and a half behind the absolute

Cutting Edge of the big models and so we

have this dual challenge like how do you

hold centralized power accountable when

the existing mechanism is basically a

little bit broken and how do you address

this Mass proliferation issue when it's

unclear how to stop anything in Mass

proliferation on the web that's a really

big challenge what we've started to see

is self-organizing initiatives on the

part of the companies right so getting

together and agreeing to sign up

proactively to self oversight both in

terms of audit in terms of capabilities

that we won't explore

etc etc now I think that's only

partially reassuring to people clearly

maybe not even reassuring at all but the

reality is I think it's the right first

step given that we haven't actually

demonstrated the large-scale harms to

arise from AIS just yet I mean this is

one of the first occasions I think in

general purpose waves of technology that

we're actually starting to adopt a

precautionary principle I'm a big

advocate of that I think that we should

be approaching a Do no harm principle

and that may mean that we have to leave

some of the benefits on the tree and

some fruit may just not be picked for a

while and we might lose some gains over

a couple of years where we may look back

in hindsight and think oh well we could

have actually gone a little bit faster

there I think that's the right trade-off

this is a moment of caution things are

accelerating extremely quickly and we

can't yet do the balance between the

harms and benefits perfectly well until

we see how this wave unfolds a little

bit so I like the fact that our company

inflection Ai and the other big

developers are trying to take a little

bit more of a cautious approach I think

that's a really interesting point

because you know we are having this

conversation you have written both of

you you extensively about the challenges

posed by this technology there's now a

parlor game amongst you know

practitioners in this world about you

know what is the risk of extinction

level events where there's a huge amount

of talk about this and I don't know in

fact I should probably ask you what

percentage of your time probably right

now it's you know close to 100 of your

time is focused on the risk since you're

promoting your book but it's it is

there's a lot of attention on this which

is which is good

um we are thinking about it early so

that gets us I think now to the most

important part of our conversation which

is what do we do and you Mustafa you lay

out a 10-point plan which is you know

the kind of action do kind of thing that

uh that someone who doesn't just comment

like you and I do but actually does

things we do so tell us what do we need

to do as as Humanity as governments as

societies to ensure that we capture the

gains from this technology but we

minimize the risks there are some very

practical things I mean so for example

red teaming these models means

adversarially testing them and trying to

put them under as much pressure as

possible to push them to generate advice

for example on how to generate a

biological or chemical weapon how to

create a bomb for example or even push

them to be very sexist racist biased in

some way and that already is pretty

significant we can see their weaknesses

I mean part of the release of these

models in the last year has given

everybody I think the opportunity to see

not just how good they are but also

their weaknesses and that is reassuring

we need to do this out in the open

that's why I'm a huge fan of the open

source Community as it is at the moment

because real developers get to play with

the models and actually see how hard it

is to produce the capabilities that

sometimes I think we fear that they're

just going to be super manipulative and

persuasive and you know destined to be

awful so that's the first thing is doing

it out in the open the second thing is

that we have to share the best practices

and so there's a competitive tension

there because safety is going to be an

asset you know I'm gonna deliver a

better product to my consumers if I have

a safer model but of course there's got

to be a requirement that if I discover a

vulnerability a weakness in the model

then I should share that just as we have

done for actually decades in many waves

of Technology not just in software

security for example but in Flight

Aviation you know the Black Box recorder

for example if there's a significant

incident not only does it record all the

Telemetry on board the aircraft but also

everything that the pilots say in the

cockpit and if there's a significant

safety incident then that's shared all

around the world with all of the

competitors which is great aircrafts are

one of the safest ways to get around

despite you know on the face of it if

you described it to an alien being 40

000 feet in the sky is a very strange

thing to do so I think there's precedent

there that we can we can follow

um

I do also agree that is probably time

for us to explicitly declare that we

should not allow these tools to be used

for electioneering I mean we cannot

trust them yet we cannot trust them to

be stable and reliable we cannot allow

people to be using them for counterfeit

digital people and clearly we've talked

about that already so there are some

capabilities which we can start to take

off the table another one would be

autonomy right right now I think

autonomy is a pretty dangerous set of

methods it's exciting it represents a

possibility that could be truly

incredible but we haven't wrapped our

hands around what the risks and

limitations are likewise training an AI

to update and improve its own code this

notion of recursive self-improvement

right closing the loop so that the AI is

in charge of defining its own goals

acquiring more resources updating its

own code with respect to some objective

these are pretty dangerous capabilities

just as we have kyc know your customer

or just as we license development

developers of nuclear technologies and

all the component involved in that

supply chain there'll be a moment where

if some of the big technology you know

providers want to experiment with those

capabilities then they should expect

there to be robust audits you know they

should expect them to be licensed and

there should be independent oversight so

how do you get that done and there seem

to be there is there are several

challenges in doing it one is the

division between the relatively few

Leading Edge models of which you have

won and then the larger tale of Open

Source models where the you know the

ability to build the model is

decentralized lots of people have access

to it my sense is that the capabilities

of the latter are a little bit behind

the capabilities of the former but they

are growing all the time and so if you

have

really considerable open source

capability what is not to stop the angry

teenager in some small town developing

capabilities that could shut down the

local hospital and how do you in your

regulatory framework guard against that

well look part of the challenge is that

these models are getting smaller and

more efficient and we know that from the

history of Technologies anything that is

useful and valuable to us gets cheaper

easier to use and it proliferates far

and wide so the destiny of this

technology over a two three four decade

period has to be proliferation and we

have to confront that reality it isn't a

contradiction to name the fact that

proliferation seems to be inevitable but

containing centralized power is an

equivalent challenge

so there is no easy answer to that I

mean Beyond surveilling the internet it

is pretty clear that in 30 years time

like you say garage tinkerers will be

able to experiment if you look at the

trajectory on synthetic biology we now

have have desk desktop synthesizers that

is the ability to engineer new synthetic

compounds they cost about twenty

thousand dollars and they basically

enable you to create potentially

molecules which are you know more

transmissible or more lethal than we had

with covid you can basically experiment

and the challenge there is that there's

no oversight you buy it off the shelf

you don't need a great deal of training

probably an undergraduate in biology

today and you'll be able to experiment

now of course they're going to get

smaller easier to use and spread far and

wide and so my book I'm really trying to

popularize the idea that this is the

defining containment challenge of the

next few decades so you use the word

containment which is interesting because

you know I'm sure the word containment

with you brings immediately you know

inspires images of George Cannon and and

you know the post-war Cold War Dynamic

and we're now you know we're in a

geopolitical world now that whether or

not you call it a new cold war is one of

great tension between the US and China

can this kind of containment as as

Mustafa calls it be done when you have

the sort of tensions you've got between

the world's big players are the you know

is the right Paradigm thinking about the

arms control treaties of the Cold War

like how do we go about doing this at a

kind of international level I think this

is the biggest problem that if it was a

question of you know humankind versus a

common threat of these new intelligent

alien agents here on Earth then yes I

think there are ways we can contain them

but if the humans are divided among

themselves

and are in an arms race then it's

because it becomes almost impossible to

contain this alien intelligence and and

there is I I'm tending to think of it

more in in terms of of really an alien

invasion

that like somebody coming and telling us

that you know there is a fleet an alien

Fleet of spaceships coming from planet

Zircon or whatever with with highly

intelligent beings they'll be here in

five years and take over the planet

maybe they'll be nice maybe they'll

solve cancer and climate change but we

are not sure

this is what we are facing except that

the aliens are not coming in spaceships

from planet Zircon that are coming from

the Laboratories

the actual characterization of the

nature of the technology an alien has by

default agency these are going to be

tools that we can apply we have narrow

settings yes but let's say they have

they potentially have agency we can try

to prevent them from having agency but

we know that they are going to be highly

intelligent and at least potentially

have agency and this is a very very

frightening mix something we never

confronted before again atom bombs

didn't have a potential for agency

printing presses did not have a

potential for agency this thing again

unless we contain it and the problem of

content is very difficult because

potentially they'll be more intelligent

than us how do you prevent something

more intelligent than you from become

from developing the agency it has I'm

not saying it's impossible I'm just

saying it it's very very difficult

I think our best bet is not to kind of

think in terms of some kind of rigid

regulation you should do this you

shouldn't do that it's in developing new

institutions

living institutions that are capable of

understanding the very fast developments

and reacting on the fly at present the

problem is that the only institutions

who really understand what is happening

are the institutions who develop the

technology the governments most of them

seem quite clueless about what's

happening also universities I mean the

amount of talent and the amount of the

the economic resources in the private

sector is far far higher than in the

universities so and again I'm I

appreciate that there are actors in the

private sector like Mustafa who are

thinking very seriously about regulation

and containment but we must have an

external entity

in in the game and for that we need to

develop new institutions that will have

the human resources that will have the

the economic and technological resources

and also will have the public trust

because without public trust it won't

work are we capable of creating such new

institutions I don't know I do think Eva

Rays is an important point which is as

we started this conversation and you

were painting the picture of five years

time and you were saying that the AIS

would be ubiquitous we'd all have our

own ones but that they would have the

capability to act not just to process

information they would have the

creativity they have now and the ability

to act but already from these generative

AI models the power that we've seen in

the last year two three four years has

been that they have been able to act in

ways that you and your other your fellow

technologists didn't anticipate they

they reached you know you didn't

anticipate you know the the speed with

which they would you would Win It Go or

so forth there was a the Striking thing

about them is that they have developed

in

unanticipatedly fast ways so if you

combine that with capability you don't

have to go as far as Yuval is saying and

saying that they're all more intelligent

than humans but there is an

unpredictability there that I think does

raise the concerns that Uval raises

which is you their creators can't quite

predict what powers they will have

they may not be fully autonomous but

they will be moving some ways towards

there and so how do you guard against

that or how do you you know red teaming

you use the phrase which is that I

understand it is that you know you keep

checking what's happening and tweak them

when you've seen what's when you

pressure test them you try to make them

fit you can't pressure test for

everything in advance so there is a I

think a very real point that Yuval is

making about as the capabilities

increase

so the risks increase of relying on you

and other Creator companies to to make I

mean it's a very fair question and

that's why I've long been calling for

the precautionary principle we should

both take some capabilities off the

table and classify those as high risk I

mean frankly the EU AI act which has

been in draft for three and a half years

is very sensible has a risk-based

framework that applies to each

application domain whether it's

Healthcare or self-driving or facial

recognition and it basically takes

certain capabilities off the table when

that threshold is exceeded I listed a

few earlier autonomy for example it's

clearly a capability that it has the

potential to be high risk recursive

self-improvement the same story so this

is the moment when we have to adopt a

precautionary principle not through any

fear-mongering but just as a logical

sensible way to proceed another model

which I think is very sensible is to

take an ipcc style approach an

international consensus around an

investigatory power to establish the

scientific fact basis for where we are

with respect to capabilities and that

has been an incredibly valuable process

set aside the negotiation and the policy

making just the evidence observing where

are we you don't have to take it from me

you should be able to take an

independent panel of experts who I would

personally Grant access to everything in

my company if they were a trusted true

impartial actor without question we

would Grant complete access and I know

that many of the other companies would

do the same again people are drawn

towards the kind of of scenario of the

AI creates a lethal virus Ebola plus

kovid and kills everybody let's go in

the more Economist Direction Financial

systems like you gave as a new touring

test the idea of AI making money what's

wrong with making money wonderful thing

so let's say that you have an AI which

has a better understanding of the

financial system than most humans most

politicians maybe most Bankers

and uh let's think back to the

2007-2008 financial crisis it started

with this I was about they called CDO

cdus this is exactly something that

these genius mathematicians invented

nobody understood them except for a

handful of Genius mathematicians in Wall

Street which is why nobody regulated

them and almost nobody saw the financial

crash coming what happens again this

kind of of apocalyptic scenario which

you don't see in Hollywood science

fiction movies the AI invents a new

class of financial devices that nobody

understands it's beyond human capability

to understand it's such complicated math

so much data nobody understands it it

makes billions of dollars billions and

billions of dollars and then it brings

down the world economy and no human

being understand what the hell is

happening like the prime ministers the

presidents the the financial ministers

what what is happening and again this is

not fantastic I mean we saw it with

human mathematicians in 2007-8 I think

that's that look that's one you know you

you can easily paint paint pictures here

that make you want to jump off the

nearest cliff and you know that's that's

one but actually my other response to

mustafa's laying out of where you say

well we just need to rule out certain

actions is to go back to the geopolitics

is it sensible for a country to rule out

certain capabilities if the other side

is not going to rule them out so you

have a you have a kind of political

economy problem going down the road that

you learn we this is a moment when we uh

collectively in the west have to

establish our values and stand behind

them what we cannot have is a race to

the bottom that says just because

they're doing it we should take the same

risk if we adopt that approach and cut

Corners left right and Center we'll

ultimately pay the price and that's not

an answer to well they're going to go

off and do it anyway we've said only

seen that with lethal autonomous weapons

I mean there's been a negotiation in the

U.N to regulate lethal autonomous

weapons for over 20 years and they

barely reached agreement on the

definition the definition of lethal

autonomous weapons let alone any

consensus so that's not great but we do

have to accept that it's the inevitable

trajectory and from our own perspective

we have to decide what we're prepared to

tolerate in society with respect to free

acting AIS facial surveillance facial

recognition and you know generally

autonomous systems I mean so far we've

taken a pretty cautious approach when we

don't have drones flying around

everywhere we can already it's totally

possible technically to autonomously fly

a drone to navigate around London we've

we've ruled it out right we don't yet

have autonomous self-driving cars even

though you know with some degree of harm

they are actually pretty well

functioning so the regulatory process is

also a cultural process of what we think

is socially and politically acceptable

at any given moment and I think an

appropriate level of caution is is what

we're seeing

much but I completely agree on that that

we need in many fields the Coalition of

the willing and if some actors in the

world don't want to join it's it's in

our interest

so again something like Banning Bots

impersonating people so some countries

will not agree but that doesn't matter

to protect our societies it's still a

very good idea to have these kinds of

regulations so that area of agreement is

one to bring us to a close but I want to

end by asking both of you and use first

Mustafa you are you know both raising

alarms but you are heavily involved in

creating this future

why do you carry on I personally believe

that it is possible to get the upsides

and minimize the downsides in the AI

that we have created Pi which stands for

personal intelligence is one of the

safest in the world today it doesn't

produce the racist toxic bias greeds

that they did two years ago it doesn't

fall victim to any of the jailbreaks The

Prompt hacks the adversarial red teams

none of those work and we've made safety

an absolute number one priority in the

design of our product so my goal has

been to do my very best to demonstrate a

path forward in the best possible way

this is an inevitable unfolding over

multiple decades this really is

happening the coming wave is coming and

I think my contribution is to try to

demonstrate in the best way that I can a

manifestation of a personal intelligence

which really does adhere to the best

safety constraints that we could

possibly think of so you've all you've

you've heard mustafa's explanation for

why he continues you look back over

human history now as you look forward is

this a technology and a pace of

innovation that Humanity will come to

regret or should Mustafa carry on it

could be gonna I can't predict the

future I would say that we invest so

much in developing artificial

intelligence and we haven't seen

anything yet like it's it's still the

very first baby steps of artificial

intelligence in terms of like you think

about I don't know the evolution of

organic life this is like the amoeba of

artificial intelligence and it won't

take millions of years to get to T-Rex

maybe it will take 20 years to get to

T-Rex and but one thing to remember is

that we also our own minds have a huge

scope for development uh also with

Humanity we haven't seen our full

potential yet and if we invest for every

dollar and minute that we invest in

artificial intelligence we invest

another dollar a minute in developing

our own Consciousness our own mind I

think we'll be okay but but I don't see

it happening I don't see this kind of

investment in in human beings that we

are seeing in in the machine well for me

this conversation with the two of you

has been just that investment thank you

both very much indeed thank you thank

you thank you

foreign

foreign

foreign