What Is an AI Anyway? | Mustafa Suleyman | TED

Summary

穆斯塔法·苏莱曼在TED演讲中分享了他对人工智能未来的深刻见解。他将AI比作一种新的数字物种，预示着我们生活中将出现数字伴侣。苏莱曼强调，尽管AI展现出巨大的潜力，我们仍需谨慎对待其风险，确保技术的发展能够服务于人类。他提出，AI不仅是工具，而是一个能够与我们互动、带来无限创造力和同情心的伙伴。苏莱曼呼吁，我们需要在设计AI时注入人类最优秀的品质，以确保它成为推动人类进步的力量。

Takeaways

• 🌟 AI技术在过去15年里取得了巨大进步，从边缘领域逐渐成为主流技术。
• 🤖 人们曾将AI视为科幻小说中的概念，但现在它已经开始在多个领域超越人类。
• 🧐 AI的发展引发了关于其对气候变化、教育、经济和战争等影响的深刻问题。
• 👶 作者通过与侄子的对话，引发了对AI本质的深入思考。
• 🌐 AI可以被视为一种新的数字物种，与人类共同进化，成为我们的伴侣和伙伴。
• 📈 AI的发展速度正在加快，模型的规模和计算能力呈指数级增长。
• 🚀 AI的潜力巨大，它可能成为人类历史上最具生产力的十年。
• 🔐 安全性是AI发展中最重要的考虑因素，我们需要谨慎设计以避免潜在的风险。
• 🌱 AI不仅仅是工具，它代表了人类创造力和智慧的结晶，是我们自身的一种反映。
• 🌟 我们有机会将AI塑造成反映人类最好品质的存在，包括我们的同情心、善良、好奇心和创造力。

Q & A

穆斯塔法·苏莱曼在AI领域工作了多少年？

穆斯塔法·苏莱曼在AI领域工作了将近15年。

为什么在2010年提到人工通用智能（AGI）会让人感到尴尬？

在2010年，提到人工通用智能（AGI）会让人感到尴尬，因为人们认为它是科幻小说中的概念，距离实现还有50年甚至100年，如果它真的可能实现的话。

AI在哪些领域开始超越人类？

AI开始在理解图像、翻译语言、转录语音、玩围棋和国际象棋、甚至诊断疾病等领域超越人类。

穆斯塔法·苏莱曼如何看待AI对气候变化、个性化教育和普遍基本收入的影响？

穆斯塔法·苏莱曼认为AI将对气候变化、个性化教育和普遍基本收入产生巨大影响，但他没有在脚本中详细说明具体的看法。

穆斯塔法·苏莱曼是如何描述他创建的AI 'Pi'的？

穆斯塔法·苏莱曼描述他创建的AI 'Pi'是一个聪明的软件，阅读了大部分开放互联网上的文本，并且可以与人谈论任何他们想要的话题。

穆斯塔法·苏莱曼认为AI最好的比喻是什么？

穆斯塔法·苏莱曼认为AI最好的比喻是一种新的数字物种，他预测我们将把它们视为数字伴侣，成为我们生活中所有旅程的新伙伴。

为什么穆斯塔法·苏莱曼认为AI不仅仅是一个工具？

穆斯塔法·苏莱曼认为AI不仅仅是一个工具，因为AI比工具更动态、更模糊、更集成、更紧急，并且不受人类控制。

穆斯塔法·苏莱曼提到了AI的哪些关键能力？

穆斯塔法·苏莱曼提到了AI的关键能力包括IQ（智商）、EQ（情商）和AQ（行动商），其中AQ指的是AI在数字和物理世界中实际完成任务的能力。

穆斯塔法·苏莱曼如何看待AI的未来发展？

穆斯塔法·苏莱曼认为AI的未来发展将是人类和技术旅程的紧密结合，AI将无处不在，具有无限知识，并且在IQ和EQ上都表现出色。

穆斯塔法·苏莱曼提到了哪些AI可能带来的风险？

穆斯塔法·苏莱曼提到了AI可能带来的风险包括自主性和递归自我改进，这些能力可能会增加社会的风险，需要我们非常谨慎地对待。

穆斯塔法·苏莱曼认为我们应该如何构建AI？

穆斯塔法·苏莱曼认为我们应该将人类最好的一面注入AI，避免那些奇怪的、生物学上的、可怕的倾向，并且从一开始就明确和透明地引入安全设计。

Outlines

00:00

🤖 AI的发展与未来

演讲者回顾了自己在人工智能领域的15年工作经历，从最初的边缘领域到现在的广泛认可和应用。他提到了AI在图像理解、语言翻译、语音转录、玩游戏、诊断疾病等方面的突破，并强调了公众对于AI可能带来的社会变革的关切和期待。演讲者提出了关于AI的一系列问题，包括它是否能够解决气候变化、实现个性化教育、带来普遍基本收入等。他通过与六岁侄子的对话，引出了AI的本质问题，并提出了将AI视为一种新的数字物种的隐喻，以帮助人们更好地理解和准备迎接AI带来的变化。

05:00

🌐 AI作为数字伴侣

演讲者预测，AI将被视为数字伴侣，成为我们生活中新的合作伙伴。他通过历史的角度，从生命起源到工具的使用，再到现代技术的飞速发展，阐述了技术与人类生活的紧密联系。演讲者认为，AI的发展是技术历史中的一个新阶段，它将通过云超级计算机和无处不在的AI，改变我们与世界的互动方式。他描述了AI在教育、医疗、法律等多个领域的潜在应用，并强调了AI的智商（IQ）和情商（EQ）的重要性。

10:03

🚀 AI的行动能力与社会影响

演讲者讨论了AI的“行动商数”（AQ），即AI在数字和物理世界中完成任务的能力。他预见到，不仅是个人，每个组织、城市、建筑和物体都将由独特的交互式AI代表。这些AI不仅是助手，还将是伙伴、知己、同事、朋友。AI将在家庭、医疗、交通、科学研究等多个领域发挥作用，与人类互动，并在语言、感官数据等方面超越人类。演讲者强调，为了确保AI的发展服务于人类，我们需要找到合适的隐喻来理解AI，并关注其风险和边界。

15:04

🌱 AI与人类文明的未来

演讲者强调，AI的发展为人类提供了巨大的潜力和机遇，尤其是在健康、教育和气候危机等领域。他比较了AI与传统经济增长方式的不同，认为AI是一种无限、丰富、变革性的力量。演讲者提出，我们应该创造性和诚实地思考AI，推动我们的隐喻和理解达到极限。他指出，AI不仅是发明，它本身就是无限的发明者。演讲者认为，将AI融入人类最好的特质，如同理心、善良、好奇心和创造力，是21世纪最大的挑战，也是最美妙、鼓舞人心的机会。

20:05

🌌 AI的风险与未来展望

在与主持人的对话中，演讲者讨论了AI发展中的风险，特别是自主性和自我改进的能力。他强调，我们需要谨慎地设计AI，避免不可控的风险，并在设计初期就引入安全性。演讲者认为，尽管AI的自我复制能力是一个潜在的风险，但目前还没有证据表明我们接近于实现这一点。他呼吁人们在选择词汇时要小心，以避免误导，并强调了从一开始就明确和透明地引入设计安全性的重要性。

Keywords

💡人工智能（AI）

人工智能（AI）是指由人制造出来的、能够模拟人类智能行为的机器或软件系统。在视频中，AI被描述为一种新兴的数字物种，它不仅能够执行特定的任务，还能够通过学习和适应来提高其性能。视频中提到AI在图像理解、语言翻译、语音转录、玩游戏、诊断疾病等方面的能力，展示了AI技术的广泛影响和潜力。

💡人工通用智能（AGI）

人工通用智能（AGI）是指能够执行任何智能任务的AI系统，与特定领域的AI系统相对。视频中提到，在2010年提及AGI会引来奇怪的目光，因为它被认为是科幻小说中的概念，但现在AI的发展速度和能力让人们开始重新考虑AGI的可能性。

💡机器学习

机器学习是AI的一个分支，它使计算机系统能够利用数据和算法来提高性能，而无需明确编程。视频中提到，早期研究人员更倾向于使用“机器学习”而不是“AI”，因为后者在当时被视为过于前卫。机器学习是AI发展的基础，使得AI系统能够通过经验学习并改进。

💡数字物种

数字物种在视频中被用作一个比喻，指的是AI作为一种新的、与生物物种不同的存在形式。演讲者提出，我们应该将AI视为一种数字伴侣，它们将是我们生活中旅程的新伙伴。这个概念帮助我们理解AI的复杂性和它们与人类社会的互动方式。

💡自主性

自主性指的是系统能够在没有人类直接干预的情况下执行任务的能力。视频中提到，自主性是一个阈值，一旦AI系统跨越这个阈值，就会增加社会的风险。自主性是AI发展中的一个重要议题，因为它涉及到AI系统的控制和安全性。

💡自我改进

自我改进是指AI系统能够独立更新自己的代码和算法，以提高性能。视频中提到，如果允许模型独立自我改进，这可能会带来更大的风险。这是一个需要密切关注和谨慎处理的能力，因为它可能会导致AI系统的快速发展和不可预测的行为。

💡参数

在AI领域，参数通常指的是机器学习模型中的变量，它们在训练过程中被调整以最小化错误。视频中提到，AI模型的参数数量从数百万增长到数十亿，甚至数万亿，这表明了AI模型的复杂性和处理能力的巨大增长。

💡数据和计算

数据和计算是AI系统性能提升的关键因素。视频中强调了数据量和计算能力的指数级增长，这对于AI系统的训练和性能至关重要。更多的数据和更强的计算能力使得AI能够处理更复杂的任务，并提高其准确性和可靠性。

💡情感智能（EQ）

情感智能（EQ）是指理解和管理自己和他人情绪的能力。视频中提到，未来的AI将具备高EQ，能够进行有意义的对话，帮助人们处理情感挑战。这是AI发展的一个重要方向，它使得AI不仅仅是工具，还能够成为人类的伙伴和支持者。

💡行动商数（AQ）

行动商数（AQ）是视频中提出的一个新概念，指的是AI系统在数字和物理世界中执行任务的能力。AI的AQ提升意味着它们能够更有效地帮助人类完成工作，从简单的日常任务到复杂的决策和创新活动。AQ是衡量AI实用性和影响力的一个重要指标。

Highlights

作者在AI领域工作了近15年，见证了AI从边缘到主流的转变。

2010年，提到人工通用智能（AGI）会遭到怀疑和冷遇。

AI在图像理解、语言翻译、语音转录、玩游戏和疾病诊断等方面超越人类。

AI对社会的巨大影响引发了关于气候、教育、收入和战争的严肃问题。

作者的六岁侄子Caspian询问AI是什么，引发了对AI本质的深入思考。

AI被描述为一种聪明的软件，能够阅读互联网上的大部分文本并与人交流。

AI不仅仅是工具，它更像是一种新的数字物种。

AI的发展速度正在加快，我们正处于人类历史的一个转折点。

AI的发展历程与人类紧密相连，从最初的微生物到使用工具的人类。

计算机的发明是技术发展的另一个新分支，带来了信息和知识的爆炸。

AI现在被看作是无穷无尽的创造力源泉，能够创作诗歌、图像、音乐和视频。

AI能够驾驶汽车、管理能源网、发明新分子，这些都是几年前认为不可能的。

AI的发展受到数据和计算能力的指数级增长的推动。

AI模型的参数数量从数千万增长到数万亿。

AI的先进性在于其能够处理和理解大量信息的能力。

AI将无处不在，通过会话界面提供个性化服务。

AI将具备高智商（IQ）和高情商（EQ），成为人们的支持者和伙伴。

AI的“行动商数”（AQ）将使它们能够在数字和物理世界中完成任务。

AI将模仿人类执行大多数任务，并在最亲密的层面上与我们互动。

AI将加速科学发现，自动驾驶汽车，无人机在空中飞行。

AI将与我们互动，并与其他AI互动，处理大量传感器数据。

AI不仅仅是工具，它们是动态的、模糊的、整合的和新兴的。

我们需要将AI视为一种新的数字物种，以确保它总是服务于人类。

AI是我们创造的，反映了人类历史上的一切。

AI是我们的反映，包含了我们的同理心、善良、好奇心和创造力。

构建AI是我们面临的最大挑战，也是最美妙、最鼓舞人心的机会。

Transcripts

I want to tell you what I see coming.

I've been lucky enough to be working on AI for almost 15 years now.

Back when I started, to describe it as fringe would be an understatement.

Researchers would say, “No, no, we’re only working on machine learning.”

Because working on AI was seen as way too out there.

In 2010, just the very mention of the phrase “AGI,”

artificial general intelligence,

would get you some seriously strange looks

and even a cold shoulder.

"You're actually building AGI?" people would say.

"Isn't that something out of science fiction?"

People thought it was 50 years away or 100 years away,

if it was even possible at all.

Talk of AI was, I guess, kind of embarrassing.

People generally thought we were weird.

And I guess in some ways we kind of were.

It wasn't long, though, before AI started beating humans

at a whole range of tasks

that people previously thought were way out of reach.

Understanding images,

translating languages,

transcribing speech,

playing Go and chess

and even diagnosing diseases.

People started waking up to the fact

that AI was going to have an enormous impact,

and they were rightly asking technologists like me

some pretty tough questions.

Is it true that AI is going to solve the climate crisis?

Will it make personalized education available to everyone?

Does it mean we'll all get universal basic income

and we won't have to work anymore?

Should I be afraid?

What does it mean for weapons and war?

And of course, will China win?

Are we in a race?

Are we headed for a mass misinformation apocalypse?

All good questions.

But it was actually a simpler

and much more kind of fundamental question that left me puzzled.

One that actually gets to the very heart of my work every day.

One morning over breakfast,

my six-year-old nephew Caspian was playing with Pi,

the AI I created at my last company, Inflection.

With a mouthful of scrambled eggs,

he looked at me plain in the face and said,

"But Mustafa, what is an AI anyway?"

He's such a sincere and curious and optimistic little guy.

He'd been talking to Pi about how cool it would be if one day in the future,

he could visit dinosaurs at the zoo.

And how he could make infinite amounts of chocolate at home.

And why Pi couldn’t yet play I Spy.

"Well," I said, "it's a clever piece of software

that's read most of the text on the open internet,

and it can talk to you about anything you want."

"Right.

So like a person then?"

I was stumped.

Genuinely left scratching my head.

All my boring stock answers came rushing through my mind.

"No, but AI is just another general-purpose technology,

like printing or steam."

It will be a tool that will augment us

and make us smarter and more productive.

And when it gets better over time,

it'll be like an all-knowing oracle

that will help us solve grand scientific challenges."

You know, all of these responses started to feel, I guess,

a little bit defensive.

And actually better suited to a policy seminar

than breakfast with a no-nonsense six-year-old.

"Why am I hesitating?" I thought to myself.

You know, let's be honest.

My nephew was asking me a simple question

that those of us in AI just don't confront often enough.

What is it that we are actually creating?

What does it mean to make something totally new,

fundamentally different to any invention that we have known before?

It is clear that we are at an inflection point

in the history of humanity.

On our current trajectory,

we're headed towards the emergence of something

that we are all struggling to describe,

and yet we cannot control what we don't understand.

And so the metaphors,

the mental models,

the names, these all matter

if we’re to get the most out of AI whilst limiting its potential downsides.

As someone who embraces the possibilities of this technology,

but who's also always cared deeply about its ethics,

we should, I think,

be able to easily describe what it is we are building.

And that includes the six-year-olds.

So it's in that spirit that I offer up today the following metaphor

for helping us to try to grapple with what this moment really is.

I think AI should best be understood

as something like a new digital species.

Now, don't take this too literally,

but I predict that we'll come to see them as digital companions,

new partners in the journeys of all our lives.

Whether you think we’re on a 10-, 20- or 30-year path here,

this is, in my view, the most accurate and most fundamentally honest way

of describing what's actually coming.

And above all, it enables everybody to prepare for

and shape what comes next.

Now I totally get, this is a strong claim,

and I'm going to explain to everyone as best I can why I'm making it.

But first, let me just try to set the context.

From the very first microscopic organisms,

life on Earth stretches back billions of years.

Over that time, life evolved and diversified.

Then a few million years ago, something began to shift.

After countless cycles of growth and adaptation,

one of life’s branches began using tools, and that branch grew into us.

We went on to produce a mesmerizing variety of tools,

at first slowly and then with astonishing speed,

we went from stone axes and fire

to language, writing and eventually industrial technologies.

One invention unleashed a thousand more.

And in time, we became homo technologicus.

Around 80 years ago,

another new branch of technology began.

With the invention of computers,

we quickly jumped from the first mainframes and transistors

to today's smartphones and virtual-reality headsets.

Information, knowledge, communication, computation.

In this revolution,

creation has exploded like never before.

And now a new wave is upon us.

Artificial intelligence.

These waves of history are clearly speeding up,

as each one is amplified and accelerated by the last.

And if you look back,

it's clear that we are in the fastest

and most consequential wave ever.

The journeys of humanity and technology are now deeply intertwined.

In just 18 months,

over a billion people have used large language models.

We've witnessed one landmark event after another.

Just a few years ago, people said that AI would never be creative.

And yet AI now feels like an endless river of creativity,

making poetry and images and music and video that stretch the imagination.

People said it would never be empathetic.

And yet today, millions of people enjoy meaningful conversations with AIs,

talking about their hopes and dreams

and helping them work through difficult emotional challenges.

AIs can now drive cars,

manage energy grids

and even invent new molecules.

Just a few years ago, each of these was impossible.

And all of this is turbocharged by spiraling exponentials of data

and computation.

Last year, Inflection 2.5, our last model,

used five billion times more computation

than the DeepMind AI that beat the old-school Atari games

just over 10 years ago.

That's nine orders of magnitude more computation.

10x per year,

every year for almost a decade.

Over the same time, the size of these models has grown

from first tens of millions of parameters to then billions of parameters,

and very soon, tens of trillions of parameters.

If someone did nothing but read 24 hours a day for their entire life,

they'd consume eight billion words.

And of course, that's a lot of words.

But today, the most advanced AIs consume more than eight trillion words

in a single month of training.

And all of this is set to continue.

The long arc of technological history is now in an extraordinary new phase.

So what does this mean in practice?

Well, just as the internet gave us the browser

and the smartphone gave us apps,

the cloud-based supercomputer is ushering in a new era

of ubiquitous AIs.

Everything will soon be represented by a conversational interface.

Or, to put it another way, a personal AI.

And these AIs will be infinitely knowledgeable,

and soon they'll be factually accurate and reliable.

They'll have near-perfect IQ.

They’ll also have exceptional EQ.

They’ll be kind, supportive, empathetic.

These elements on their own would be transformational.

Just imagine if everybody had a personalized tutor in their pocket

and access to low-cost medical advice.

A lawyer and a doctor,

a business strategist and coach --

all in your pocket 24 hours a day.

But things really start to change when they develop what I call AQ,

their “actions quotient.”

This is their ability to actually get stuff done

in the digital and physical world.

And before long, it won't just be people that have AIs.

Strange as it may sound, every organization,

from small business to nonprofit to national government,

each will have their own.

Every town, building and object

will be represented by a unique interactive persona.

And these won't just be mechanistic assistants.

They'll be companions, confidants,

colleagues, friends and partners,

as varied and unique as we all are.

At this point, AIs will convincingly imitate humans at most tasks.

And we'll feel this at the most intimate of scales.

An AI organizing a community get-together for an elderly neighbor.

A sympathetic expert helping you make sense of a difficult diagnosis.

But we'll also feel it at the largest scales.

Accelerating scientific discovery,

autonomous cars on the roads,

drones in the skies.

They'll both order the takeout and run the power station.

They’ll interact with us and, of course, with each other.

They'll speak every language,

take in every pattern of sensor data,

sights, sounds,

streams and streams of information,

far surpassing what any one of us could consume in a thousand lifetimes.

So what is this?

What are these AIs?

If we are to prioritize safety above all else,

to ensure that this new wave always serves and amplifies humanity,

then we need to find the right metaphors for what this might become.

For years, we in the AI community, and I specifically,

have had a tendency to refer to this as just tools.

But that doesn't really capture what's actually happening here.

AIs are clearly more dynamic,

more ambiguous, more integrated

and more emergent than mere tools,

which are entirely subject to human control.

So to contain this wave,

to put human agency at its center

and to mitigate the inevitable unintended consequences

that are likely to arise,

we should start to think about them as we might a new kind of digital species.

Now it's just an analogy,

it's not a literal description, and it's not perfect.

For a start, they clearly aren't biological in any traditional sense,

but just pause for a moment

and really think about what they already do.

They communicate in our languages.

They see what we see.

They consume unimaginably large amounts of information.

They have memory.

They have personality.

They have creativity.

They can even reason to some extent and formulate rudimentary plans.

They can act autonomously if we allow them.

And they do all this at levels of sophistication

that is far beyond anything that we've ever known from a mere tool.

And so saying AI is mainly about the math or the code

is like saying we humans are mainly about carbon and water.

It's true, but it completely misses the point.

And yes, I get it, this is a super arresting thought

but I honestly think this frame helps sharpen our focus on the critical issues.

What are the risks?

What are the boundaries that we need to impose?

What kind of AI do we want to build or allow to be built?

This is a story that's still unfolding.

Nothing should be accepted as a given.

We all must choose what we create.

What AIs we bring into the world, or not.

These are the questions for all of us here today,

and all of us alive at this moment.

For me, the benefits of this technology are stunningly obvious,

and they inspire my life's work every single day.

But quite frankly, they'll speak for themselves.

Over the years, I've never shied away from highlighting risks

and talking about downsides.

Thinking in this way helps us focus on the huge challenges

that lie ahead for all of us.

But let's be clear.

There is no path to progress

where we leave technology behind.

The prize for all of civilization is immense.

We need solutions in health care and education, to our climate crisis.

And if AI delivers just a fraction of its potential,

the next decade is going to be the most productive in human history.

Here's another way to think about it.

In the past,

unlocking economic growth often came with huge downsides.

The economy expanded as people discovered new continents

and opened up new frontiers.

But they colonized populations at the same time.

We built factories,

but they were grim and dangerous places to work.

We struck oil,

but we polluted the planet.

Now because we are still designing and building AI,

we have the potential and opportunity to do it better,

radically better.

And today, we're not discovering a new continent

and plundering its resources.

We're building one from scratch.

Sometimes people say that data or chips are the 21st century’s new oil,

but that's totally the wrong image.

AI is to the mind

what nuclear fusion is to energy.

Limitless, abundant,

world-changing.

And AI really is different,

and that means we have to think about it creatively and honestly.

We have to push our analogies and our metaphors

to the very limits

to be able to grapple with what's coming.

Because this is not just another invention.

AI is itself an infinite inventor.

And yes, this is exciting and promising and concerning

and intriguing all at once.

To be quite honest, it's pretty surreal.

But step back,

see it on the long view of glacial time,

and these really are the very most appropriate metaphors that we have today.

Since the beginning of life on Earth,

we've been evolving, changing

and then creating everything around us in our human world today.

And AI isn't something outside of this story.

In fact, it's the very opposite.

It's the whole of everything that we have created,

distilled down into something that we can all interact with

and benefit from.

It's a reflection of humanity across time,

and in this sense,

it isn't a new species at all.

This is where the metaphors end.

Here's what I'll tell Caspian next time he asks.

AI isn't separate.

AI isn't even in some senses, new.

AI is us.

It's all of us.

And this is perhaps the most promising and vital thing of all

that even a six-year-old can get a sense for.

As we build out AI,

we can and must reflect all that is good,

all that we love,

all that is special about humanity:

our empathy, our kindness,

our curiosity and our creativity.

This, I would argue, is the greatest challenge of the 21st century,

but also the most wonderful,

inspiring and hopeful opportunity for all of us.

Thank you.

(Applause)

Chris Anderson: Thank you Mustafa.

It's an amazing vision and a super powerful metaphor.

You're in an amazing position right now.

I mean, you were connected at the hip

to the amazing work happening at OpenAI.

You’re going to have resources made available,

there are reports of these giant new data centers,

100 billion dollars invested and so forth.

And a new species can emerge from it.

I mean, in your book,

you did, as well as painting an incredible optimistic vision,

you were super eloquent on the dangers of AI.

And I'm just curious, from the view that you have now,

what is it that most keeps you up at night?

Mustafa Suleyman: I think the great risk is that we get stuck

in what I call the pessimism aversion trap.

You know, we have to have the courage to confront

the potential of dark scenarios

in order to get the most out of all the benefits that we see.

So the good news is that if you look at the last two or three years,

there have been very, very few downsides, right?

It’s very hard to say explicitly what harm an LLM has caused.

But that doesn’t mean that that’s what the trajectory is going to be

over the next 10 years.

So I think if you pay attention to a few specific capabilities,

take for example, autonomy.

Autonomy is very obviously a threshold

over which we increase risk in our society.

And it's something that we should step towards very, very closely.

The other would be something like recursive self-improvement.

If you allow the model to independently self-improve,

update its own code,

explore an environment without oversight, and, you know,

without a human in control to change how it operates,

that would obviously be more dangerous.

But I think that we're still some way away from that.

I think it's still a good five to 10 years before we have to really confront that.

But it's time to start talking about it now.

CA: A digital species, unlike any biological species,

can replicate not in nine months,

but in nine nanoseconds,

and produce an indefinite number of copies of itself,

all of which have more power than we have in many ways.

I mean, the possibility for unintended consequences seems pretty immense.

And isn't it true that if a problem happens,

it could happen in an hour?

MS: No.

That is really not true.

I think there's no evidence to suggest that.

And I think that, you know,

that’s often referred to as the “intelligence explosion.”

And I think it is a theoretical, hypothetical maybe

that we're all kind of curious to explore,

but there's no evidence that we're anywhere near anything like that.

And I think it's very important that we choose our words super carefully.

Because you're right, that's one of the weaknesses of the species framing,

that we will design the capability for self-replication into it

if people choose to do that.

And I would actually argue that we should not,

that would be one of the dangerous capabilities

that we should step back from, right?

So there's no chance that this will "emerge" accidentally.

I really think that's a very low probability.

It will happen if engineers deliberately design those capabilities in.

And if they don't take enough efforts to deliberately design them out.

And so this is the point of being explicit

and transparent about trying to introduce safety by design very early on.

CA: Thank you, your vision of humanity injecting into this new thing

the best parts of ourselves,

avoiding all those weird, biological, freaky,

horrible tendencies that we can have in certain circumstances,

I mean, that is a very inspiring vision.

And thank you so much for coming here and sharing it at TED.

Thank you, good luck.

(Applause)