Geoffrey Hinton in conversation with Fei-Fei Li — Responsible AI development

Arts & Science - University of Toronto
23 Feb 2024108:12

Summary

TLDR在多伦多的MaRS Discovery District举行的一场激动人心的对话中,多伦多大学的Geoffrey Hinton教授和斯坦福大学的Fei-Fei Li教授共同探讨了人工智能(AI)的未来及其对社会的影响。作为深度学习领域的先驱,Hinton和Li分享了他们对于AI技术快速发展的看法,特别是聊天机器人技术如何在短时间内成为公众关注的焦点。他们深入讨论了AI在教育、医疗、农业等领域的应用潜力,同时也对AI带来的风险,包括就业、假信息传播、自动化武器等问题进行了分析。最后,他们呼吁对AI进行负责任的开发和应用,以确保技术进步能够造福全人类。

Takeaways

  • 🧠 Geoffrey Hinton和Fei-Fei Li讨论了人工智能的当前状态和未来展望。
  • 🔍 他们讨论了大型语言模型(LLM)和基础模型的潜力和挑战。
  • 🌐 Fei-Fei Li强调了人文主义在人工智能发展中的重要性,呼吁在技术创新中考虑人类福祉。
  • 🤖 Geoffrey Hinton表达了对超智能带来的潜在风险的担忧,特别是它可能对人类构成的威胁。
  • 🏛️ 他们讨论了公共部门和学术界在人工智能研究和开发中的作用,以及私营部门的巨大影响力。
  • 📚 强调了数据的重要性,以及大数据如何推动深度学习和人工智能的进步。
  • 💡 讨论了人工智能在解决全球性问题(如健康、教育和气候变化)中的应用潜力。
  • 🤝 强调了跨学科合作的重要性,以及如何通过结合多个领域的专长来克服人工智能领域的挑战。
  • 📈 他们探讨了未来人工智能技术的发展趋势,以及如何平衡创新与伦理和社会责任。
  • 🚀 讨论了启发年轻一代科学家和工程师追求在人工智能领域的大胆和有意义的工作的重要性。

Q & A

  • Meric Gertler 在致辞中提到了哪些土地的传统守护者?

    -Meric Gertler 提到了 Huron-Wendat、Seneca 和 Mississaugas of the Credit 作为多伦多大学所在地的传统守护者。

  • Geoffrey Hinton 被誉为什么?

    -Geoffrey Hinton 被誉为深度学习的教父。

  • Fei-Fei Li 在斯坦福大学担任什么角色?

    -Fei-Fei Li 是计算机科学的首席 Sequoia 教授,并且是人类中心 AI 研究所的联合主任。

  • Schwartz Reisman 创新园区的建立是由谁使之成为可能?

    -Schwartz Reisman 创新园区的建立是由 Heather Reisman 和 Gerry Schwartz 的慷慨和远见使之成为可能。

  • ImageNet 数据集是由谁创立的?

    -ImageNet 数据集是由 Fei-Fei Li 创立的。

  • AlexNet ImageNet 挑战的背后主要贡献者是谁?

    -AlexNet ImageNet 挑战的背后主要贡献者包括 Ilya Sutskever、Alex Krizhevsky 和 Geoffrey Hinton。

  • ImageNet 挑战对计算机视觉领域产生了什么影响?

    -ImageNet 挑战极大地推动了计算机视觉领域的发展,证明了深度神经网络在图像识别任务中的超凡效果。

  • 2012年 ImageNet 挑战的胜利如何影响了神经网络的接受度?

    -2012年 ImageNet 挑战的胜利显著提高了神经网络的接受度,让以前批评和怀疑神经网络的研究人员开始采用这种技术。

  • Fei-Fei Li 的书《我所见的世界》主要讲述了什么?

    -Fei-Fei Li 的书《我所见的世界》主要讲述了她对于大数据的理解、深度学习的潜力以及人工智能技术的道德责任。

  • Geoffrey Hinton 对于人工智能未来的看法是怎样的?

    -Geoffrey Hinton 对于人工智能未来持有担忧态度,特别是关于超智能带来的存在风险和道德挑战。

Outlines

00:00

😀 开场致辞

Meric Gertler 在多伦多大学的 MaRS Discovery District 举办的 Radical AI 创始人活动上致开幕辞,欢迎来宾,并简要介绍活动背景。他强调了多伦多大学在人工智能领域的领导地位,特别提到 Geoffrey Hinton 和他的学生对深度学习的贡献。Gertler 随后介绍了即将推出的 Schwartz Reisman Innovation campus,强调其将如何推动创新和科技发展。

05:01

🤖 AI技术的进步与影响

活动中讨论了 Geoffrey Hinton 和 Fei-Fei Li 在人工智能领域的贡献,特别是在深度学习和计算机视觉方面。提到了他们的工作对学术界和工业界的影响,以及他们在推动技术创新和理解人工智能潜力方面的角色。此外,还强调了对于技术伦理、社会责任以及AI对未来人类社会影响的关注。

10:05

🌐 ImageNet的诞生与影响

Fei-Fei Li 讲述了 ImageNet 数据库的创建过程,包括最初的挑战、目标和对机器学习领域的深远影响。ImageNet 通过提供大量标注图片,极大地推动了计算机视觉和深度学习技术的发展。这一段还提及了与 ImageNet 相关的重要事件,如 ImageNet Challenge,以及这些活动是如何促进深度学习模型,特别是卷积神经网络在图像识别任务中的突破性进展。

15:05

💡 AI技术的转折点

本段回顾了 AlexNet 在 ImageNet 挑战中的胜利以及其对计算机视觉领域影响的描述。讨论了这一事件是如何标志着深度学习和神经网络在图像识别中取得领先地位的转折点,以及它是如何引发学术界和工业界对深度学习技术的广泛关注和投资。

20:06

📚 从学术到实践

深入讨论了 AI 技术从学术研究到实际应用的转变,特别是在语音识别和图像识别方面。通过具体案例,如 Geoffrey Hinton 和他的学生在语音识别领域的工作,以及后来这些技术如何被应用到实际产品中,例如 Google 的 Android 系统。

25:09

🚀 ChatGPT的兴起

讨论了 OpenAI 发布 ChatGPT 及其对大众的影响,特别是如何让广泛的用户群体直接体验到 AI 技术的强大能力。ChatGPT 被视为一个使 AI 技术大众化的关键时刻,引发了对人工智能潜力和风险的广泛讨论。

30:11

🤝 AI技术的社会责任

本段强调了 AI 技术发展的社会责任和伦理考量。Geoffrey Hinton 和 Fei-Fei Li 讨论了他们对 AI 技术可能带来的正面和负面影响的看法,包括就业、隐私、歧视和偏见等问题。他们强调了需要采取措施来确保 AI 技术的发展符合人类的最佳利益。

35:16

📈 AI未来的展望

在对话的最后部分,Geoffrey Hinton 和 Fei-Fei Li 分享了他们对 AI 未来发展的展望,包括技术潜力、社会影响以及需要解决的关键挑战。他们强调了跨学科合作的重要性,以及未来研究人员在推动技术进步的同时,也需关注其对社会的影响。

Mindmap

Keywords

💡人工智能

人工智能是指由人造系统所表现出来的智能,包括学习、理解、推理、规划等能力。在视频中,人工智能是核心话题,涉及深度学习、机器学习和其他技术的发展,以及这些技术如何推动科技和社会进步。

💡深度学习

深度学习是一种通过使用具有多层结构的神经网络来模拟人脑分析和学习的方法。在视频中,深度学习被认为是推动人工智能发展的关键技术之一,特别是在图像识别、语言处理等领域的应用。

💡ImageNet

ImageNet是一个大规模的图像数据库,旨在使用这些图像来训练机器视觉系统。在视频中,ImageNet被提到是因为其对于深度学习和视觉识别技术的发展起到了重要作用,尤其是通过举办挑战赛推动了神经网络研究。

💡基准测试

基准测试在人工智能领域是用来评估模型性能的一种方法,通过对特定数据集的处理能力进行评价。视频中提到基准测试,特别是ImageNet挑战赛,作为评估和推动人工智能技术进步的重要手段。

💡聊天机器人

聊天机器人是一种能够模拟人类对话进行交流的人工智能应用程序。视频中,ChatGPT作为一个例子,展示了人工智能在理解和生成自然语言方面的先进能力。

💡社会影响

社会影响指的是技术,尤其是人工智能技术,对社会、经济和文化等方面产生的影响。视频中讨论了人工智能的积极影响,如促进科学研究,以及潜在的负面影响,如职业被替代和伦理问题。

💡伦理

伦理在人工智能领域涉及确保技术发展和应用不会伤害人类或社会的原则和标准。视频中强调了在人工智能研究和应用中考虑伦理问题的重要性,特别是在数据隐私、偏见和控制风险方面。

💡责任感

责任感在视频中被提出作为人工智能开发者和研究者在推进技术时应持有的一种态度,意味着他们应当对其创造的技术及其对社会的影响负责。

💡可解释性

可解释性关注于使人工智能系统的决策过程更加透明和可理解。视频中提到,在开发更复杂的人工智能系统时,提高其可解释性是一大挑战,同时也是赢得公众信任的关键。

💡转型风险

转型风险指的是由于技术创新而导致的经济和社会结构变化可能带来的风险。视频中探讨了人工智能带来的转型风险,如就业市场变化和社会不平等的加剧,以及如何通过政策和教育减轻这些风险。

Highlights

Geoffrey Hinton and Fei-Fei Li discuss the evolution and impact of AI, highlighting its potential to transform various sectors.

Acknowledgement of the land by Meric Gertler, emphasizing the University of Toronto's commitment to recognizing its history.

The significance of the MaRS Discovery District and the University of Toronto's role in the AI community is highlighted.

Geoffrey Hinton's pivotal role in deep learning and AI, earning him the nickname 'Godfather of Deep Learning'.

Fei-Fei Li's contributions to AI, particularly in computer vision and her leadership at the Human-Centered AI Institute.

The collaboration between academia and industry in advancing AI research and innovation.

The introduction of the Schwartz Reisman Innovation campus as a hub for AI thought leadership.

Discussion on the dual nature of AI: its incredible potential for innovation and the growing concerns over its societal impact.

The story of the ImageNet database creation by Fei-Fei Li and its monumental impact on AI research.

The breakthrough moment of AlexNet in the ImageNet competition, showcasing the power of deep neural networks.

The transformative effect of AI and deep learning on fields such as drug discovery, medical diagnostics, and more.

Concerns and ethical considerations about AI shaping the future of humanity.

The conversation shifts towards addressing the urgency of responsible AI development and regulation.

Both speakers reflect on their personal journeys into AI research, highlighting their motivations and challenges.

The discussion emphasizes the importance of collaboration between academia, industry, and government in shaping the future of AI.

The audience engagement through questions reflects the community's interest in understanding and contributing to responsible AI.

Transcripts

play00:03

Well, good afternoon everyone.

play00:06

Got to love the buzz in the room here today.

play00:10

Welcome to the MaRS Discovery District,

play00:13

this wonderful complex

play00:16

for this very special radical AI founders event,

play00:19

co-hosted by the University of Toronto.

play00:21

My name is Meric Gertler

play00:22

and it's my great privilege to serve as president

play00:25

of the University of Toronto.

play00:27

Before we begin, I want to acknowledge the land

play00:30

on which the University of Toronto operates.

play00:34

For thousands of years, it has been the traditional land

play00:35

of the Huron-Wendat, the Seneca,

play00:38

and the Mississaugas of the Credit.

play00:40

Today, this meeting place is still the home

play00:42

to many Indigenous people from across Turtle Island,

play00:45

and we are very grateful to have the opportunity

play00:47

to work and to gather on this land.

play00:51

Well, I'm truly delighted to welcome you all

play00:53

to this discussion between Geoffrey Hinton,

play00:56

university professor Emeritus at the University of Toronto,

play00:59

known to many as the Godfather of Deep Learning,

play01:02

and Fei-Fei Li the inaugural Sequoia Professor

play01:05

in Computer Science at Stanford University,

play01:08

where she's Co-Director of Human-centered AI Institute.

play01:12

I want to thank Radical Ventures

play01:14

and the other event partners for joining with U of T

play01:17

to create this rare and special opportunity.

play01:22

Thanks in large part to the groundbreaking work

play01:24

of Professor Hinton and his colleagues,

play01:27

the University of Toronto has been at the forefront

play01:29

of the academic AI community for decades.

play01:33

Deep learning is one of the primary breakthroughs

play01:36

propelling the AI boom,

play01:39

and many of its key developments were pioneered

play01:41

by Professor Hinton and his students at U of T.

play01:45

This tradition of excellence,

play01:46

this long tradition continues into the present.

play01:50

Our faculty, students and graduates,

play01:53

together with partners at the Vector Institute

play01:56

and at universities around the world

play01:58

are advancing machine learning and driving innovation.

play02:03

Later this fall, our faculty, staff, students,

play02:06

and partners will begin moving into phase one

play02:09

of the beautiful new Schwartz Reisman Innovation campus

play02:12

just across the street.

play02:13

You may have noticed a rather striking building

play02:16

at the corner with the official opening plan

play02:19

for early next year.

play02:21

This facility will accelerate innovation and discovery

play02:25

by creating Canada's largest

play02:27

university based innovation hub,

play02:30

made possible by a generous and visionary gift

play02:33

from Heather Reisman and Gerry Schwartz.

play02:36

The innovation campus will be a focal point

play02:38

for AI thought leadership,

play02:40

hosting both the Schwartz Reisman Institute

play02:42

for Technology and Society,

play02:44

led by Professor Gillian Hadfield

play02:47

and the Vector Institute.

play02:50

It is already clear that artificial intelligence

play02:53

and machine learning are driving innovation

play02:55

and value creation across the economy.

play02:59

They're also transforming research

play03:01

in fields like drug discovery, medical diagnostics,

play03:04

and the search for advanced materials.

play03:07

Of course, at the same time,

play03:09

there are growing concerns over the role

play03:11

that AI will play in shaping humanity's future.

play03:15

So today's conversation clearly addresses

play03:17

a timely and important topic,

play03:19

and I am so pleased that you have all joined us

play03:22

on this momentous occasion.

play03:24

So without further ado,

play03:25

let me now introduce today's moderator, Jordan Jacobs.

play03:29

Jordan is managing partner

play03:31

and co-founder of Radical Ventures,

play03:33

a leading venture capital firm

play03:35

supporting AI-based ventures here in Toronto

play03:37

and around the world.

play03:39

Earlier he co-founded Layer 6 AI

play03:41

and served as Co-CEO prior to its acquisition

play03:45

by TD Bank Group, which he joined as Chief AI Officer.

play03:49

Jordan serves as a Director of the Canadian Institute

play03:52

for Advanced Research,

play03:54

and he was among the founders of the Vector Institute,

play03:57

a concept that he dreamed up

play03:58

with Tomi Poutanen, Geoff Hinton,

play04:00

Ed Clark, and a few others.

play04:02

So distinguished guests,

play04:04

please join me in welcoming Jordan Jacobs.

play04:08

(audience applauding)

play04:16

- Come on up.

play04:17

Thanks very much, Meric.

play04:18

I wanted to start by thanking a number of people

play04:21

who've helped to make this possible today,

play04:24

University of Toronto and Meric,

play04:26

Melanie Woodin, Dean of Arts and Science,

play04:28

and a number of partners that have brought this to fruition.

play04:32

So this is the first in our annual four part series

play04:36

of founder AI masterclasses that we run at Radical.

play04:40

This is the third year we've done it,

play04:41

and today's the first one of this year.

play04:44

We do it in person and online.

play04:46

So we've got thousands of people watching this online.

play04:48

So if you decide you need to start coughing,

play04:50

maybe head outside.

play04:54

We do that in partnership with the Vector Institute

play04:57

and thank them very much

play04:59

for their participation and support

play05:01

with the Alberta Machine Intelligence Institute in Alberta,

play05:04

and with Stanford AI, thanks to Fei-Fei.

play05:08

So thank you all of you for being excellent partners.

play05:11

We're hoping that this is gonna be

play05:12

a really interesting discussion.

play05:14

This is the first time that Geoff and Fei-Fei,

play05:16

who I like to think of as friends, and I get to talk to,

play05:19

but this is the first time

play05:20

they're doing this publicly together.

play05:21

So it's, I think, gonna be

play05:22

a really interesting conversation.

play05:24

Let me quickly do some deeper explanations

play05:29

of their background.

play05:31

Geoff is often called the Godfather

play05:33

of Artificial Intelligence.

play05:34

He's won the touring award.

play05:37

He is a Professor Emeritus University of Toronto,

play05:39

co-founder of the Vector Institute,

play05:41

also mentored in a lot of the people

play05:45

who have gone on to be leaders in AI globally,

play05:47

including at the big companies

play05:49

and many of the top research labs in the world in academia.

play05:52

So when we say godfather, it really is,

play05:56

there are many kinds of children and grandchildren of Geoff

play06:00

who are leading the world in AI

play06:02

and that all comes back to Toronto.

play06:05

Fei-Fei is the founding Director of the Stanford Institute

play06:07

for Human-Centered AI,

play06:10

Professor at Stanford.

play06:12

She's an elected member of the National Academy

play06:15

of Engineering in the US,

play06:16

the National Academy of Medicine

play06:18

in the American Academy of Arts and Science.

play06:20

During a sabbatical from Stanford in 2017/18,

play06:23

she stepped in for a role as a Vice-President at Google

play06:26

as Chief Scientist of AI/ML at Google Cloud.

play06:30

There's many, many other things we could say about Fei-Fei

play06:32

but she also has an amazing number of students

play06:35

who have gone on to be leaders in the field globally.

play06:38

And really importantly,

play06:41

and so for those of you who haven't heard yet,

play06:43

Fei-Fei has a book coming out in a couple of weeks.

play06:46

It is called, it's coming out on November 7th,

play06:49

it's called "The Worlds I See,

play06:51

Curiosity, Exploration, and Discovery at the Dawn of AI."

play06:54

I've read it, it's fantastic.

play06:57

You should all go out and buy it.

play06:58

I'll read you the back cover slip that Geoff wrote

play07:03

'cause it's much better than what I could say about it.

play07:07

So here's Geoff's description.

play07:10

"Fei-Fei Li was the first computer vision researcher

play07:12

to truly understand the power of big data,

play07:15

and her work opened the floodgates for deep learning.

play07:17

She delivers an urgent, clear-eyed account

play07:19

of the awesome potential

play07:21

and danger of the AI technology that she helped to unleash.

play07:24

And her call for action and collective responsibility

play07:26

is desperately needed at this pivotal moment in history."

play07:29

So I urge you all to go and pre-order the book

play07:32

and read it as soon as it comes out.

play07:34

With that, thanks Fei-Fei and Geoff for joining us.

play07:39

- Thank you, Jordan.

play07:40

(audience applauding)

play07:46

- Okay, so I think it's not an exaggeration

play07:50

to say that without these two people,

play07:52

the modern age of AI does not exist,

play07:54

certainly not in the way that it's played out.

play07:56

So let's go back to what I think is the big bang moment.

play07:59

AlexNet ImageNet, maybe Geoff,

play08:02

do you want to take us through from your perspective

play08:04

that moment which is 11 years ago now?

play08:06

- Okay, so in 2012, two of my very smart graduate students

play08:12

won a competition, a public competition,

play08:14

and showed that deep neural networks

play08:17

could do much better than the existing technology.

play08:21

Now, this wouldn't have been possible

play08:23

without a big data set that you could train them on.

play08:26

Up to that point, there hadn't been a big data set

play08:28

of labeled images,

play08:30

and Fei-Fei was responsible for that data set.

play08:33

And I'd like to start by asking Fei-Fei

play08:37

whether there were any problems

play08:38

in putting together that data set?

play08:40

(audience laughing)

play08:41

- Well, thank you Geoff, and thank you Jordan,

play08:44

and thank you University Toronto for this,

play08:47

it's really fun to be here.

play08:49

So yes, the data set that Geoff you're mentioning

play08:51

is called ImageNet.

play08:53

And I began building it 2007

play08:57

and spent the next three years pretty much

play09:01

with my graduate students building it.

play09:03

And you asked me was there a problem building it,

play09:06

where do I even begin?

play09:08

(Fei-Fei laughing)

play09:10

Even at the conception of this project

play09:13

I was told that it really was a bad idea.

play09:19

I was a young Assistant Professor.

play09:22

I remember it was my first year actually

play09:24

as a Assistant Professor at Princeton

play09:26

and for example, a very respected mentor

play09:31

of mine in the field,

play09:34

if you know the academic jargon,

play09:35

these are the people who will be writing

play09:38

my tenure evaluations, actually told me

play09:42

really out of their good heart

play09:45

that please don't do this after I told them

play09:48

what this plan is back in 2007.

play09:51

- So that would've been Jitendra right?

play09:52

(audience laughing)

play09:56

- The advice was that,

play10:00

"You might have trouble getting tenure if you do this."

play10:04

And then I also tried to invite other collaborators

play10:10

and nobody in machine learning or AI

play10:14

wanted to even go close to this project,

play10:18

and of course no funding. - Sorry.

play10:21

(audience laughing)

play10:23

- Just describe ImageNet to us

play10:26

for the people who are not familiar with what it was.

play10:28

- Yeah, so ImageNet was conceived around 2006, 2007,

play10:32

and the reason I conceived ImageNet was actually twofold.

play10:38

One is that,

play10:42

and Geoff, I think we share similar background,

play10:44

I was trained as a scientist,

play10:46

to me, doing science is chasing after North Stars.

play10:49

And in the field of AI, especially visual intelligence,

play10:52

for me, object recognition,

play10:55

the ability for computers

play10:57

to recognize there's a table in the picture

play11:00

or there's a chair is called object recognition,

play11:02

has to be a North star problem in our field.

play11:06

And I feel that we need to really put a dent

play11:10

in this problem.

play11:11

So I want to define that North Star problem,

play11:13

that was one aspect of ImageNet.

play11:16

Second aspect of ImageNet was recognizing

play11:19

that machine learning was really going in circles

play11:21

a little bit at that time,

play11:22

that we were making really intricate models

play11:27

without the kind of data to drive the machine learning.

play11:31

Of course, in our jargon,

play11:33

it's really the generalization problem, right?

play11:36

And I recognize that we really need to hit a reset,

play11:42

and rethink about machine learning

play11:43

from a data driven point of view.

play11:45

So I wanted to go crazy

play11:47

and make a data set that no one has ever seen

play11:50

in terms of its quantity and diversity and everything.

play11:56

So ImageNet after three years was a curated data set

play12:00

of internet images that's totaled

play12:04

15 million images across 22,000 concepts,

play12:11

object category concepts.

play12:13

And that was the data set

play12:17

- Just for comparison,

play12:18

at the same time in Toronto

play12:20

we were making a data set called CIFAR-10

play12:24

that had 10 different classes and 60,000 images,

play12:27

and it was a lot of work,

play12:30

with general generously paid for by CIDAR

play12:32

at five cents an image.

play12:35

- And so you turn the data set into a competition,

play12:38

just walk us through a little bit

play12:40

of what that meant,

play12:41

and then we'll kind of fast forward to 2012.

play12:44

- Right.

play12:45

So we made the data set in 2009.

play12:48

We barely made it into a poster in a academic conference.

play12:54

And no one paid attention.

play12:57

So it was a little desperate at that time.

play12:59

And I believe this is the way to go.

play13:02

And we open sourced it,

play13:04

but even with open source,

play13:08

it wasn't really picking up.

play13:11

So my students and I thought,

play13:14

well let's get a little more drive up the competition.

play13:18

Let's create a competition

play13:20

to invite the worldwide research community

play13:23

to participate in this problem of object recognition

play13:26

through ImageNet.

play13:27

So we made a ImageNet competition

play13:30

and the first feedback we got

play13:33

from our friends and colleagues is, it's too big.

play13:38

And at that time you can not fit it into a hard drive,

play13:42

let alone memory.

play13:44

So we actually created a smaller data set

play13:47

called the ImageNet challenge data set,

play13:49

which is only 1 million images

play13:53

across 1000 categories instead of 22,000 category,

play13:57

and that was unleashed in 2010, I think.

play14:03

You guys noticed it in 2011, right?

play14:06

- Yes. - Yeah.

play14:07

- And so in my lab we already had deep neural networks

play14:12

working quite well for speech recognition.

play14:15

And then Ilya Sutskever said,

play14:18

"What we've got really ought to be able

play14:19

to win the ImageNet competition."

play14:22

And he tried to convince me that we should do that.

play14:25

And I said, well, you know, it's an awful lot of data.

play14:28

And he tried to convince his friend Alex Krizhevsky,

play14:33

and Alex wasn't really interested.

play14:36

So Ilya actually pre-processed all the data

play14:39

to put it in just the form Alex needed it in.

play14:41

- You shrunk the size of the images.

play14:42

- Yes. - Yeah.

play14:44

- He shrunk the images a bit.

play14:45

- Yeah, I remember.

play14:47

- And got it pre-processed just right for Alex,

play14:49

and then Alex eventually agreed to do it.

play14:51

Meanwhile, in Yann LeCun's lab in New York,

play14:54

Yann was desperately trying to get his students

play14:56

and postdocs to work on this data set.

play14:59

'Cause he said, "The first person

play15:00

to apply convolutional nets to this data that's gonna win."

play15:03

And none of his students were interested.

play15:05

They were all busy doing other things.

play15:07

And so Alex and Ilya got on with it,

play15:09

and we discovered

play15:12

by running on the previous year's competition

play15:15

that we were doing much better than the other techniques.

play15:18

And so we knew we were gonna win the 2012 competition.

play15:21

And then there was this political problem,

play15:24

which is we thought if we showed that neural networks

play15:28

win this competition,

play15:30

the Computer Vision people,

play15:32

Jitendra in particular will say,

play15:36

well that just shows it's not a very good data set.

play15:39

So we had to get them to agree ahead of time

play15:42

that if we won the competition,

play15:44

we'd proved that neural networks worked.

play15:45

So actually called up Jitendra

play15:48

and we talked about data sets we might run on.

play15:50

And my objective was to get Jitendra to agree

play15:55

that if we could do ImageNet,

play15:57

then neural nets really worked.

play15:59

And after some discussion

play16:01

and him telling me to do other data sets,

play16:03

we eventually agreed, okay, if we could do ImageNet

play16:06

then we'd have shown neural nets work.

play16:09

Jitendra remembers it as he suggested ImageNet

play16:12

and he was the one who told us to do it,

play16:14

but it was actually a bit the other way round.

play16:17

And we did it and it was amazing.

play16:20

We got just over half the error rate

play16:22

of the standard techniques.

play16:23

And the standard techniques have been tuned for many years

play16:25

by very good researchers.

play16:27

- I remember standard technique at that time,

play16:30

the previous year is support vector machine

play16:33

with sparsification. - Right.

play16:35

- That was, so you guys submitted your competition results,

play16:41

I think it was late August or early September.

play16:46

And I remember either getting a phone call,

play16:49

or getting an email late one evening from my students

play16:54

who was running this

play16:55

because we hold the test data

play16:58

we were running on the server side.

play17:00

The goal is that we have to process all the entries

play17:04

so that we select the winners,

play17:06

and then by, I think it was beginning of October that year

play17:12

that Computer Vision Fields International Conference,

play17:18

ICCV 2012 was happening in Florence, Italy.

play17:23

We already booked a workshop,

play17:25

annual workshop at the conference.

play17:27

We will be announcing the winner,

play17:30

it's the third year.

play17:32

So a couple of weeks before we have to process the teams.

play17:38

Because it was the third year

play17:40

and frankly the previous two years

play17:42

results didn't excite me,

play17:44

and I was a nursing mother at that time.

play17:47

So I decided not to go to the third year,

play17:50

so I didn't book any tickets.

play17:52

I'm just like, too far from me.

play17:55

And then the results came in,

play17:57

that evening, phone call or email,

play18:00

I really don't remember, came in.

play18:03

And I remember saying to myself, darn it Geoff,

play18:08

now I have to get a ticket to Italy.

play18:11

Because I knew that was a very significant moment,

play18:15

especially with a convolutional neural network,

play18:17

which I learned as a graduate student,

play18:19

as a classic algorithm.

play18:22

And of course by that time

play18:24

there was only middle seats economy class

play18:28

flying from San Francisco to Florence

play18:32

with a one stop layover.

play18:34

So it was a grueling trip to go to Florence-

play18:39

- I'm sorry. - But I wanted to be there.

play18:42

(audience laughing)

play18:43

Yeah, but you didn't come.

play18:45

- No

play18:46

(audience laughing)

play18:50

Well, it was a grueling trip.

play18:52

- But did you know that would be a historical moment?

play18:55

- Yes, I did actually.

play18:58

- You did, and you still didn't come.

play18:59

But you sent Alex.

play19:01

- Alex, yes. - Yeah.

play19:02

- Who ignored all your advice?

play19:04

- Who ignored my email for multiple times,

play19:07

'cause I was like, Alex, this is so cool,

play19:09

please do this visualization, this visualization.

play19:13

He ignored me.

play19:14

But Yann LeCun came and it was because,

play19:18

for those of you who have attended

play19:20

these academic conference workshops

play19:22

tend to book these smaller rooms.

play19:24

We booked a very small room,

play19:26

probably just the middle section here.

play19:29

And I remember Yann had to stand in the back of the room

play19:33

because it was really packed,

play19:36

and Alex eventually showed up

play19:39

'cause I was really nervous

play19:40

that he wasn't even gonna show up.

play19:42

But as you predicted at that workshop

play19:46

ImageNet was being attacked.

play19:50

At that workshop there were people vocally attacking,

play19:55

this is a bad dataset.

play19:58

- In the room?

play19:59

- In the room .

play20:01

- During the presentation? - In the room.

play20:03

- But not Jitendra,

play20:05

'cause Jitendra has already agreed that it counted.

play20:06

- Yeah, I don't think Jitendra was in the room,

play20:07

I don't remember.

play20:08

But I remember it was such a strange moment for me

play20:13

because as a machine learning researcher,

play20:15

I knew history was in the making,

play20:19

yet ImageNet was being attacked.

play20:23

It was just a very strange,

play20:26

it was exciting moment.

play20:28

And then I had to hop in the middle seat

play20:31

to get back to San Francisco

play20:32

because then the next morning.

play20:33

- So you mentioned a few people

play20:36

that I want to come back to later.

play20:37

So Ilya who's founder and chief scientist at OpenAI,

play20:41

and Yann LeCun who subsequently went on

play20:44

to be head of AI at Facebook now Meta,

play20:47

and there's a number of other interesting people in the mix.

play20:50

But before we go forward

play20:53

and kind of see what that boom moment created,

play20:56

let's just go back for a little bit.

play20:59

Both of you started in this

play21:01

with kind of a very specific goal in mind

play21:04

that is an individual and I think a iconoclastic,

play21:07

and you had to persevere through the moments

play21:11

that you just described,

play21:12

but kind of throughout your careers.

play21:14

Can you just go back, Geoff maybe and start,

play21:16

give us a background

play21:17

to why did you want to get into AI in the first place?

play21:20

- I did psychology as an undergraduate.

play21:23

I didn't do very well at it.

play21:25

And I decided they were never going to figure out

play21:29

how the mind worked

play21:29

unless they figured out how the brain worked.

play21:32

And so I wanted to figure out how the brain worked

play21:35

and I wanted to have an actual model that worked.

play21:39

So you can think of understanding the brain

play21:42

as building a bridge.

play21:43

There's experimental data

play21:45

and things you can learn from experimental data,

play21:46

and there's things that will do the computations you want,

play21:50

things that will recognize objects.

play21:52

And they were very different.

play21:55

And I think of it as you want to build this bridge

play21:57

between the data and the competence,

play22:01

the ability to do the task.

play22:03

And I always saw myself

play22:05

as starting at the end of things that work,

play22:08

but trying to make them more and more like the brain,

play22:10

but still work.

play22:12

Other people tried to stay with things

play22:16

justified by empirical data,

play22:18

and try and have theories that might work.

play22:21

But we're trying to build that bridge

play22:23

and not many people were trying to build a bridge.

play22:25

Terry Sejnowski was trying to build a bridge

play22:27

from the other end,

play22:27

and so we got along very well.

play22:30

A lot of people doing,

play22:33

trying to do computer vision,

play22:34

just wanted something that worked,

play22:36

they didn't care about the brain.

play22:37

And a lot of people who care about the brain

play22:39

wanted to understand how neurons work and so on,

play22:41

but didn't want to think much

play22:43

about the nature of the computations.

play22:46

And I still see it as we have to build this bridge

play22:49

by getting people who know about the data

play22:51

and people who know about what works to connect.

play22:56

So my aim was always to make things that could do vision,

play23:02

but do vision in the way that people do it.

play23:05

- Okay, so we're gonna come back to that

play23:07

'cause I want to ask you about

play23:08

the most recent developments

play23:10

and how you think that they relate to the brain.

play23:12

Fei-Fei, so Geoff just to kind of put a framework

play23:16

on where you started, UK to the US to Canada,

play23:19

by mid to late '80, you come to Canada in '87,

play23:23

along that route, funding and interest in neural nets,

play23:27

and the way the approach that you're taking

play23:29

kind of goes like this,

play23:31

but I'd say mostly like this-

play23:34

- It went up and down.

play23:36

- Fei-Fei you started your life in a very different place.

play23:39

Like can you walk us through

play23:41

a little bit of how you came to AI?

play23:43

- Yeah, so I started my life in China,

play23:47

and when I was 15-year-old,

play23:49

my parents and I came to Parsippany, New Jersey.

play23:53

So I became a new immigrant

play23:55

and where I started was first English

play23:59

as second language classes,

play24:01

'cause I didn't speak the language,

play24:03

and just working in laundries,

play24:07

and restaurants and and so on.

play24:11

But I had a passion for physics.

play24:14

I don't know how it got into my head.

play24:17

And I wanted to go to Princeton

play24:20

because all I know was Einstein was there,

play24:23

and I got into Princeton,

play24:26

he wasn't there by the time I got into Princeton.

play24:29

- You're not that old. - Yeah.

play24:32

But there was a statue of him.

play24:35

And the one thing I learned in physics,

play24:39

beyond all the math and all that

play24:41

is really the audacity to ask the craziest questions,

play24:47

like the smallest particles of the atom world,

play24:53

or the boundary of space time and beginning of universe.

play24:57

And along the way I discover brain

play25:02

as a third year Roger Penrose and those books.

play25:08

Yeah, you might have opinions,

play25:10

but at least I've read those books.

play25:13

- It was probably better that you didn't.

play25:15

(audience laughing)

play25:18

- Well it at least got me interested in brain.

play25:22

And by the time I was graduating

play25:24

I wanted to ask the most audacious question as a scientist.

play25:29

And to me the absolute most fascinating audacious question

play25:33

of my generation that was 2000 was intelligence.

play25:38

So I went to Caltech to get a dual,

play25:41

pretty much a dual PhD

play25:43

in neuroscience with Christof Koch,

play25:46

and in AI with Pietro Perona.

play25:48

So I so echo Geoff, what you said about bridge

play25:52

because that five years allow me

play25:57

to work on computational neuroscience

play26:00

and look at how the mind works,

play26:03

as well as to work on the computational side,

play26:06

and try to build that computer program

play26:10

that can mimic the human brain.

play26:13

So that's my journey, it starts from physics.

play26:16

- Okay, so your journeys intersect at ImageNet 2012.

play26:20

- By the way, I met Geoff

play26:21

when I was a graduate student. - Right, I remember,

play26:23

I used to go visit Pietro's lab.

play26:25

- Yeah.

play26:26

- In fact he actually offered me a job

play26:27

at Caltech when I was 17.

play26:31

- You would've been my advisor.

play26:32

- No, I would not, not when I was 17.

play26:39

- Oh, okay.

play26:40

- Okay, so we intersected at ImageNet,

play26:42

I mean in the field everyone knows

play26:44

that ImageNet is this big bang moment

play26:46

and subsequent to that first the big tech companies come in

play26:49

and basically start buying up your students and you,

play26:55

and to get them into the companies.

play26:56

I think they were the first ones

play26:58

to realize the potential of this.

play27:01

I would like to talk about that for a moment,

play27:02

but kind of fast forwarding,

play27:04

I think it's only now since ChatGPT

play27:08

that the rest of the world is catching up

play27:09

to the power of AI.

play27:10

Because finally you can play with it.

play27:12

You can experience it,

play27:14

in the boardroom they can talk about it,

play27:16

and then go home,

play27:17

and then the 10-year-old kid

play27:19

has just written a dinosaur essay

play27:21

for fifth grade with ChatGPT.

play27:22

So that kind of transcendent experience

play27:24

of everyone being able to play with it,

play27:25

I think has been a huge shift.

play27:27

But in the period in between which is 10 years,

play27:31

there is kind of this explosive growth of AI

play27:36

inside the big tech companies,

play27:38

and everyone else is not really noticing what's going on.

play27:41

Can can you just talk us through your own experience?

play27:44

Because you experienced

play27:46

a kind of a ground zero post ImageNet.

play27:49

- It's difficult for us to get into the frame

play27:53

of everybody else not realizing what was going on,

play27:55

'cause we realized what was going on.

play27:58

So a lot of the universities

play28:02

you'd have thought would be right at the forefront

play28:04

were very slow in picking up on it.

play28:06

So MIT for example, and Berkeley,

play28:08

I remember going even talking in Berkeley

play28:11

in I think 2013

play28:16

when already AI was being very successful

play28:18

in Computer Vision.

play28:19

And afterwards a graduate student came up to me

play28:22

and he said, "I've been here like four years

play28:23

and this is the first talk

play28:24

I've heard about neural networks.

play28:25

They're really interesting."

play28:27

- Well, they should have gone to Stanford.

play28:30

- Probably, probably.

play28:33

But the same with MIT,

play28:34

they were rigidly against having neural nets.

play28:38

And the ImageNet moment started to wear them down

play28:43

and now they're big proponents of neural nets.

play28:46

But it's hard to imagine now,

play28:48

but around 2010 or 2011

play28:52

there was the Computer Vision people,

play28:56

very good Computer Vision people

play28:57

who were really adamantly against neural nets.

play29:00

They were so against it that, for example,

play29:02

one of the main journals, the IEEE PAM recognition-

play29:07

- PAM? - PAM.

play29:09

Had a policy not to referee papers

play29:11

on neural nets at one point.

play29:13

Just send them back, don't referee them,

play29:15

it's a waste of time, it shouldn't be in PAM.

play29:17

And Yann LaCun sent a paper to a conference

play29:20

where he had a neural net that was better at identifying,

play29:25

at doing segmentation of pedestrians

play29:27

than the state of the art.

play29:28

And it was rejected.

play29:30

And it was one of the reasons it was rejected

play29:33

was one of the referees said,

play29:34

"This tells us nothing about vision."

play29:36

'Cause they had this view of how computer vision works,

play29:38

which is you study the nature of the problem of vision,

play29:43

you formulate an algorithm that'll solve it,

play29:46

you figure out how to implement that algorithm,

play29:48

and then you publish a paper.

play29:50

In fact, it doesn't work to it

play29:51

- I have to defend my field, not everybody,

play29:54

- Not everybody.

play29:55

- So there are people who are-

play29:57

- But most of them were adamantly against neural nets.

play30:00

And then something remarkable happened

play30:02

after the ImageNet competition, which is,

play30:04

they all changed within about a year.

play30:08

All the people who have been the biggest critics

play30:11

of neural nets started doing neural nets,

play30:13

much to our chagrin,

play30:13

and some of them did it better than us.

play30:16

So this (indistinct) in Oxford, for example,

play30:18

made a better neural net very quickly.

play30:22

But they behaved like scientists ought to behave,

play30:24

which is that the strong belief this stuff was rubbish.

play30:27

Because of ImageNet we could eventually show

play30:30

that it wasn't and then they changed.

play30:32

So that was very comforting.

play30:34

- And just to carry it forward,

play30:35

so what you're trying to show,

play30:38

you're trying to label using the neural nets,

play30:40

these 15 million images accurately,

play30:43

you've got them all labeled in the background

play30:45

so you can measure it.

play30:47

The error rate when you did it

play30:48

dropped from 26% the year before,

play30:50

I think to 16% or so.

play30:52

- Yep. - I think it's 15.3.

play30:54

- Okay. And then it subsequently keeps-

play30:56

- 15.32.

play30:58

(audience laughing)

play30:59

- I knew you'd remember. - Which randomization?

play31:02

- Geoff doesn't forget.

play31:05

And then in subsequent years

play31:06

people are using more powerful neural nets

play31:08

and it continues to drop

play31:09

to the point where it surpasses-

play31:12

- 2015.

play31:13

So there's a Canadian,

play31:15

very smart Canadian undergrad who joined my lab,

play31:19

his name is Andrej Karpathy.

play31:22

And he got bored one summer and said,

play31:25

"I want to measure how humans do."

play31:28

So you should go read his blog.

play31:30

So he had all these like human doing image

play31:35

that test parties,

play31:37

he had to bribe them with pizza I think.

play31:40

with my students in the lab.

play31:41

And they got to a accuracy about 5%, and that-

play31:47

Was it five or 3.5?

play31:49

- Three. - Three.

play31:50

3.5 I think.

play31:51

- So humans basically make mistakes about

play31:53

3% of the time? - Right, right.

play31:55

And then I think 2016, I think a resonant passed it.

play31:59

- Yeah. - Right, it was resonant,

play32:02

is that year's winning algorithm

play32:04

passed the human performance.

play32:06

- And then ultimately you had to retire the competition

play32:08

because it was so much better than humans that had-

play32:11

- We had to retire 'cause we run out funding.

play32:14

- Okay, alright.

play32:16

It's a different reason.

play32:17

- A bad reason. - Still run outta funding

play32:19

- Instantly that student started life

play32:22

at the University of Toronto. - Yes.

play32:24

- Where he went to your lab,

play32:24

and then he went to be head of research at Tesla.

play32:29

- Okay, first of all, he came to Stanford

play32:32

to be a PhD student.

play32:34

And yesterday night we were talking,

play32:36

actually there was a breakthrough dissertation,

play32:40

in the middle of this.

play32:42

And then he became part of the founding team of OpenAI.

play32:45

- But then he went to Tesla. - And then he went to Tesla.

play32:49

- And then he thought better of it.

play32:51

- He's back.

play32:53

But I do want to answer your question of that 10 years.

play32:56

- Well there's a couple of developments along the way.

play32:58

- Right. - Transformers.

play33:00

- Right.

play33:01

- So the transformer paper is written, the research done,

play33:04

paper written inside Google,

play33:05

another Canadian is a co-author there, Aidan Gomez,

play33:08

who's now the CEO and co-founder of Cohere,

play33:12

who I think was a 20-year-old intern at Google Brain

play33:17

when co-authored the paper.

play33:19

So there's a tradition

play33:20

of Canadians being involved in these breakthroughs.

play33:23

But Geoff, you were at Google when the paper was written,

play33:27

was there an awareness inside Google

play33:30

of how important this would be?

play33:32

- I don't think there was, maybe the authors knew,

play33:35

but it took me several years

play33:37

to realize how important it was.

play33:39

And at Google people didn't realize

play33:41

how important it was until BERT

play33:43

so BERT used transformers,

play33:45

and BERT then became a lot better

play33:47

at a lot of natural language processing benchmarks

play33:50

for a lot of different tasks.

play33:52

And that's when people realized transformers were special.

play33:56

- So 2017 the transformer paper was published.

play33:59

I also joined Google,

play34:01

and I think you and I actually met

play34:03

on my first week. - Right.

play34:05

- I think most of 2017 and 2018

play34:08

was neuro-architecture search. - Right.

play34:11

- I think that was Google's bet.

play34:13

- Yep. - And there was a lot

play34:15

of GPUs being used.

play34:17

So it was a different bet.

play34:20

- So just to explain that neural architecture search

play34:23

essentially means this,

play34:25

you get yourself a whole lot of GPUs,

play34:29

and you just try lots of different architectures

play34:32

to see which works best and you automate that.

play34:34

It's basically automated evolution

play34:35

for neural net architectures.

play34:37

- It's like hyper parameter to new.

play34:39

- Yeah. - Yeah.

play34:40

- And it led to some- - Good way.

play34:41

- Quite big improvements. - Yeah.

play34:43

- But nothing like transformers.

play34:45

And transformers were a huge improvement

play34:47

for natural language

play34:48

- Neural architecture search was mostly the ImageNet.

play34:52

- Yeah. - Yeah.

play34:53

- So I'll tell you our experience of transformers.

play34:55

So we were doing our company Layer 6 at the time,

play34:57

I think we saw a pre-read of the paper

play35:01

and we were in the middle of a fundraising

play35:04

and a bunch of acquisition offers and read the paper.

play35:07

And I mean, not just me,

play35:09

but my partner told me who had studied with you,

play35:10

and Maksims Volkovs who came out of the group lab.

play35:16

And we thought this is the next iteration of neural nets,

play35:19

we should sell the company,

play35:20

start a venture fund and invest in these companies

play35:23

that are gonna be using transformers.

play35:24

So we figured it would take five years

play35:27

to get adopted beyond Google.

play35:28

And then from that moment forward,

play35:29

it would be 10 years for all the software

play35:31

in the world to get replaced

play35:32

or embedded with this technology.

play35:34

We made that decision five years and two weeks

play35:37

before ChatGPT came out.

play35:39

So I'm glad to see we were good at predicting,

play35:42

but I have to give credit to my co-founders

play35:44

who I thought I understood what the paper was,

play35:48

but they were able to explain it fully.

play35:50

- I should just correct you,

play35:51

I don't think Tomi ever studied with me.

play35:53

He wanted to come study with me,

play35:55

but a colleague in my department told him

play35:57

if he came to work with me,

play35:58

that would be the end of his career

play35:59

and he should go do something else.

play36:01

- So he took the classes,

play36:03

and this is my partner who in the late '90s

play36:08

was doing a master's at U of T,

play36:10

and he wanted to go study with Geoff, studied neural nets.

play36:13

And his girlfriend, now wife's father,

play36:16

who was a engineering professor, said, "Don't do that,

play36:19

neural nets are a dead end."

play36:21

So instead he took the classes

play36:23

but wrote his thesis in cryptocurrency.

play36:26

(audience laughing)

play36:31

Okay, so-

play36:32

- Are you still gonna talk about the 10 years?

play36:35

Because I think there's something important.

play36:37

- Yeah, so go ahead.

play36:39

- So I do think there's something important the world

play36:42

overlooked this 10 years between ImageNet, AlexNet

play36:46

and ChatGPT.

play36:49

Most of the world sees this as a tech 10 years,

play36:53

or we see it as a tech 10 years,

play36:57

in the big tech there's things brewing.

play36:59

I mean it took sequence to sequence transformer,

play37:02

but things are brewing.

play37:04

But I do think for me personally

play37:09

and for the world, it's also a transformation

play37:13

between tech to society.

play37:15

I actually think personally, I grew from a scientist

play37:19

to a humanist in this 10 years.

play37:21

Because having joined Google for that two years

play37:25

in the middle of the transformer papers,

play37:27

I begin to see the societal implication of this technology.

play37:34

It was post AlphaGo moment

play37:37

and very quickly we got to the AlphaFold moment.

play37:42

It was where bias it was creeping out,

play37:45

there was privacy issues.

play37:48

And then we're starting to see the beginning

play37:50

of disinformation and misinformation.

play37:53

And then we're starting to see the talks

play37:56

of job within a small circle,

play37:59

not within in a big public discourse.

play38:03

It was when I grew personally anxious,

play38:06

I feel, you know 2018-

play38:11

Oh, oh, it was also right after Cambridge Analytica.

play38:15

So that huge implication of technology, not AI per se,

play38:19

but it's algorithm driven technology on election,

play38:23

that's when I had to make a personal decision

play38:27

of staying at Google or come back to Stanford.

play38:30

And I knew the only reason I would come back

play38:32

to Stanford was starting this human-centered AI institute

play38:36

to really, really understand the human

play38:39

side of this technology.

play38:40

So I think this is a very important 10 years,

play38:44

even though it's kind of not in the eyes of the public,

play38:48

but this technology is starting to really creep into

play38:52

the rest of our lives.

play38:54

And of course 2022,

play38:57

it's all shown under the daylight

play39:00

how profound this is.

play39:03

- There's an interesting footnote

play39:04

to what happened during that period as well,

play39:06

which is ultimately you and Ilya and Alex joined Google,

play39:13

but before that there was a big Canadian company

play39:18

that had the opportunity to get access to this technology.

play39:20

Do you want us, I've heard this story

play39:22

but I don't think it's ever been shared publicly.

play39:24

Maybe do you want to share that story for a second?

play39:26

- Okay, so the technology that we were using

play39:29

for the ImageNet,

play39:32

we developed it in 2009 for doing speech recognition,

play39:36

for doing the acoustic modeling, bit of speech recognition.

play39:39

So you can take the sound wave

play39:41

and you can make a thing called a spectrogram,

play39:43

which just tells you at each time

play39:45

how much energy that is at each frequency.

play39:47

So you're probably used to seeing in spectrograms.

play39:50

And what you'd like to do is look at a spectrogram

play39:53

and make guesses about which part

play39:55

of which phonamium is being expressed

play39:58

by the middle frame of the spectrogram.

play40:01

And two students, George Dahl

play40:05

and another student

play40:07

who I shared with Gerald Penn called Abdo,

play40:12

he had a longer name, we all called him Abdo,

play40:17

who was a speech expert, George was a learning expert.

play40:20

Over the summer of 2009,

play40:22

they made a model that was better

play40:24

than what 30 years of speech research

play40:26

had been able to produce,

play40:29

and big, big teams working on speech research.

play40:31

And the model was slightly better,

play40:32

not as big as the ImageNet gap, but it was better.

play40:35

And that model was then ported to IBM and to Microsoft

play40:44

by George went to Microsoft

play40:47

and Abdo went to IBM,

play40:49

and those big speech groups started using neural nets then.

play40:52

And I had a third student

play40:54

who'd been working on something else,

play40:55

called Navdeep, Navdeep Jaitly.

play40:58

And he wanted to take this speech technology

play41:03

to a big company,

play41:06

but he wanted to stay in Canada

play41:07

for complicated visa reasons.

play41:09

And so we got in touch with Blackberry, RIM,

play41:14

and we said we've got this new way

play41:16

of doing speech recognition

play41:18

and it works better than the existing technology

play41:21

and we'd like a student to come to you over the summer

play41:22

and show you how to use it,

play41:24

and then you can have the best speech recognition

play41:26

in your cell phone.

play41:28

And they said after some discussions,

play41:32

a fairly senior gap Blackberry said,

play41:33

"We are not interested."

play41:35

So our attempt to give it to Canadian industry failed.

play41:41

And so then Navdeep took it to Google,

play41:45

and Google were the first to get it into a product.

play41:48

So in 2012, around the same time

play41:52

as we won the ImageNet competition,

play41:54

George and Abdo's speech recognition acoustic model,

play41:58

the acoustic model was in,

play42:00

there was a lot of work making it a good product

play42:02

and making it have low latency and so on,

play42:06

that came out in the Android.

play42:07

And there was a moment when the Android suddenly became

play42:10

as good as Siri at speech recognition

play42:12

and that was a neural net.

play42:14

And I think for the people high up in the big companies,

play42:17

that was another ingredient.

play42:18

They saw it get this dramatic result for vision,

play42:22

but they also saw that it was already out in a product

play42:24

for speech recognition was working very well there too.

play42:27

So I think that combination of it does speech,

play42:29

it does vision, clearly it's gonna do everything.

play42:33

- We won't say anymore

play42:34

about Blackberry. - It was a shame.

play42:36

It was a shame that Canadian industry didn't-

play42:39

I think we might have still had Blackberries

play42:41

if that happened.

play42:42

(audience laughing)

play42:44

- Alright, we'll leave that one there.

play42:46

(audience laughing)

play42:48

I thought it was a story, I've heard this story before,

play42:49

but I thought it was important for the rest of the world

play42:51

to know some of what went on behind the scenes,

play42:54

why this technology didn't stay in Canada

play42:56

even though it was offered for free.

play42:58

Okay, so let's advance forward.

play43:00

We now have post transformers,

play43:03

Google is starting to use this

play43:05

and develop it in a number of different ways.

play43:07

OpenAI, where your former student Ilya had left Google,

play43:11

been a founder of OpenAI with Elon Musk and Sam Altman,

play43:15

Greg Brockman and a few others.

play43:17

Ilya is the chief scientist,

play43:19

and Andrej your student as a co-founder.

play43:23

So they are working together a very small team

play43:26

to basically take turns,

play43:28

well initially the idea was we're gonna build AGI,

play43:33

artificial general intelligence,

play43:35

ultimately the transformer paper comes out,

play43:38

they start to adopt at some point transformers,

play43:41

and they start to make extraordinary gains internally,

play43:45

they're not really sharing publicly

play43:47

in what they're able to do in language understanding

play43:49

and a number of other things.

play43:52

They had efforts going on in robotics that spun out.

play43:54

Pieter Abbeel ended up spinning out Covariant,

play43:55

a company we subsequently invested in and other things.

play43:58

But so the language part of it advances,

play44:01

and advances and advances.

play44:03

People outside OpenAI don't really know

play44:05

to the extent what's going on.

play44:07

And then ChatGPT comes out November 30th last year.

play44:11

So 10 months ago. - Well, GPT-2

play44:15

caught the attention of some of us.

play44:18

I think actually, I think by the time GPT-2 came out,

play44:24

my colleague Percy Liang, an LP Professor at Stanford,

play44:29

I remember he came to me and say,

play44:31

"Fei-Fei I have a whole different realization

play44:37

of how important this technology is."

play44:39

So to the credit of Percy,

play44:41

he immediately asked HAI to set up a center to study this.

play44:47

And I don't know if this is contentious in Toronto,

play44:53

Stanford is the university that coined the term

play44:56

foundation models,

play44:58

and some people call it LLM- large language model.

play45:02

But going beyond language, we call it a foundation model.

play45:06

We created the center of research

play45:09

for foundation model before, I think before 3.5 came out.

play45:15

So definitely before ChatGPT. - Just describe

play45:18

what a foundation model is

play45:20

just for those who are not familiar.

play45:22

- That's actually a great question.

play45:24

Foundation model, some people feel

play45:26

it has to have transformer in it.

play45:28

I don't know if you use- - No, it just has to be

play45:30

a very big huge amount of data.

play45:31

- Very large, pretrained with huge amount of data.

play45:35

And I think one of the most important thing

play45:37

of a foundation model is the generalizability

play45:40

of multiple tasks.

play45:42

You're not training it for example, machine translation.

play45:46

So in NLP, machine translation is a very important task,

play45:50

but the kind of foundation model

play45:52

like GPT is able to do machine translation,

play45:55

is able to do conversation, summarization,

play45:58

and blah blah blah.

play46:00

So that's a foundation model

play46:01

and we're seeing that now in multimodality.

play46:04

We're seeing a vision, in robotics, in video and so on.

play46:07

So we created that.

play46:09

But you're right, the public sees this in the-

play46:14

- 10 months ago. - What did you say?

play46:17

- October 30th. - November, I think.

play46:18

- November.

play46:19

- One other very important thing about foundation models,

play46:22

which is for a long time in cognitive science,

play46:26

the general opinion was that these neural nets,

play46:30

if you give 'em enough training data,

play46:32

they can do complicated things,

play46:34

but they need an awful lot of training data.

play46:36

They need to see thousands of cats.

play46:40

And people are much more statistically efficient.

play46:44

That is they can learn to do these things on much less data.

play46:47

And people don't say that so much anymore

play46:49

because what they were really doing was comparing

play46:53

what an MIT undergraduate can learn to do

play46:56

on the limited amount of data

play46:58

with what a neural net that starts with random weights

play47:00

can learn to do on a limited amount of data.

play47:03

- Yeah,

play47:04

that's an unfair comparison. - And if you want to make

play47:07

a fair comparison,

play47:07

you take a foundation model that is a neural net

play47:10

that's been trained on lots and lots of stuff

play47:12

and then you give it a completely new task,

play47:13

and you ask how much data does it need

play47:15

to learn this completely new task?

play47:18

And that's called few shot learning

play47:20

'cause it doesn't take much.

play47:22

And then you discover these things

play47:25

are statistically efficient.

play47:26

That is, they compare quite favorably with people

play47:29

in how much data they need to learn to do a new task.

play47:32

So the old kind of innatist idea

play47:34

that we come with lots of innate knowledge,

play47:37

and that makes us far superior to these things,

play47:39

you just learn everything from data.

play47:41

People have pretty much given up on that now

play47:43

because you take a foundation model

play47:44

that had no innate knowledge but a lot of experience

play47:47

and then you give it a new task,

play47:49

it learns pretty efficiently.

play47:50

It doesn't need huge amounts of data.

play47:53

- You know, my PhD is in one-shot learning,

play47:56

but it's very interesting,

play47:59

even in Beijing framework you could pre-train,

play48:02

but it's only in the neuro network

play48:04

kind of pre-training really can get you this multitask.

play48:08

- Right.

play48:10

- Okay, so this basically gets productized in ChatGPT,

play48:15

the world experiences it, which is only 10 months ago,

play48:18

although for some of us

play48:19

it feels like- - Seems longer.

play48:20

- Much longer. - It feels like forever.

play48:23

- Because you suddenly you have this,

play48:25

you had this big bang that happened a long time ago

play48:28

that I think for a long time

play48:30

no one really saw the results of it,

play48:32

suddenly, I mean my comparison would be

play48:34

there's planets that are formed,

play48:36

and stars that are visible,

play48:38

and everyone can experience

play48:39

the results of what happened 10 years before,

play48:41

and then transformed, etc.

play48:42

So the world suddenly becomes very excited about

play48:47

what I think feels to a lot of people like magic.

play48:51

Something that they can touch and they can experience

play48:53

and gives them back a feedback

play48:55

in whatever way they're asking for it.

play48:57

Whether they're putting in text prompts

play48:59

and asking for an image to be created,

play49:01

or video, or texts,

play49:03

and asking for more texts to come back

play49:05

and answer things that you would never be able to expect

play49:08

and getting those unexpected answers.

play49:10

So it feels a little bit like magic.

play49:13

My personal view is that,

play49:16

we've always moved the goal line in AI.

play49:18

AI is always the thing that we couldn't do,

play49:20

it's always the magic.

play49:21

And as soon as we get there

play49:23

then we say that's not AI at all,

play49:24

or there's people around that say, that's not AI at all.

play49:26

We move the the goal line.

play49:29

In this case what was your reaction when it came out?

play49:32

I know part of your reaction is you quit Google

play49:35

and decided to do different things,

play49:37

but when you first saw it, what did you think?

play49:40

- Well, like Fei-Fei said,

play49:42

GPT-2 made a big impression on us all.

play49:45

And then there was a steady progression,

play49:47

also I'd seen things within Google before GPT-4

play49:51

and GPT-3.5 that were just as good like PaLM

play49:57

So that in itself didn't make a big effort.

play50:00

It was more PaLM made an impression on me within Google

play50:03

'cause PaLM could explain why a joke was funny,

play50:05

and I'd always just use that as a,

play50:08

we'll know that it really gets it

play50:10

when it can explain why a joke is funny.

play50:13

And PaLM could do that.

play50:15

Not for every joke but for a lot of jokes.

play50:17

- And so- - Incidentally these things

play50:18

are quite good now at explaining why jokes are funny

play50:20

but they're terrible at telling jokes,

play50:22

and there's a reason which is

play50:25

they generate text one word at a time.

play50:27

So if you ask them to tell a joke,

play50:29

what they do is they're trying to tell a joke.

play50:32

So they're gonna try and tell stuff that sounds like a joke.

play50:36

So they say, a priest and a badger went into a bar

play50:40

and that sounds a bit like the beginning of a joke

play50:43

and they keep going telling stuff

play50:45

that sounds like the beginning of a joke.

play50:47

But then they get to the point

play50:48

where they need the punchline.

play50:50

And of course they haven't thought ahead,

play50:51

they haven't thought what's going to be the punchline.

play50:53

They're just trying to make it sound like

play50:54

they lead into a joke,

play50:55

and then they give you a pathetically weak punchline,

play50:57

'cause they have to come up with some punchline.

play51:01

So although they can explain jokes

play51:03

'cause they get to see the whole joke

play51:04

before they say anything,

play51:06

they can't tell jokes, but we'll fix that.

play51:09

- Okay, so I was going to ask you

play51:10

if comedian is a job of the future or not.

play51:12

You think soon?

play51:16

- Probably not.

play51:18

- All right. - So anyway-

play51:21

- So what was your reaction to it?

play51:23

And again, you've seen things

play51:24

behind the scenes along the way.

play51:26

- A couple of reaction.

play51:28

My first reaction is of all people I thought I knew

play51:33

the power of data,

play51:35

and I was still old by the power of data.

play51:40

That was a technical reaction.

play51:42

I was like, darn it, I should have made a bigger ImageNet.

play51:45

No, but maybe not,

play51:47

but that was really- - You still could.

play51:50

- Funding is the problem.

play51:53

Yeah, so that was first.

play51:55

Second, when I saw the public awakening moment to AI

play52:00

with ChatGBT,

play52:01

not just the GPT-2 technology moment,

play52:05

I generally thought,

play52:07

thank goodness we've invested in human centered AI

play52:11

for the past four years.

play52:13

Thank goodness we have built a bridge

play52:16

with the policy makers, with the public sector,

play52:19

with the civil society.

play52:22

We have not done enough,

play52:24

but thank goodness that that conversation had started.

play52:29

We were participating it,

play52:31

we were leading some part of it.

play52:33

For example, we as a institute at Stanford,

play52:37

we're leading a critical national AI research cloud bill

play52:42

that is still going through Congress right now.

play52:47

- [Geoff] Not right now actually.

play52:51

- Senate, Senate, it's by camera,

play52:53

so at least it's moving the senate

play52:56

because we predicted the societal moment

play53:01

for this tech.

play53:02

We don't know when it would come,

play53:04

but we knew it would come,

play53:06

and it was just a sense of urgency honestly.

play53:11

I feel that this is the moment

play53:14

we really have to rise to,

play53:16

not only our passion as technologist,

play53:22

but responsibility as humanists.

play53:24

- And so you both,

play53:26

I think the common reaction of you both has been,

play53:29

we have to think about both the opportunities of this,

play53:33

but also the negative consequences of it.

play53:36

- So for me, there was something I realized

play53:38

and didn't realize until very late,

play53:40

and what got me much more interested in the societal impact

play53:44

was like Fei-Fei said, the power of data.

play53:48

These big chatbots have seen thousands

play53:52

of times more data than any person could possibly see.

play53:54

And the reason they can do that

play53:56

is 'cause you can make thousands

play53:57

of copies of the same model,

play53:59

and each copy can look at a different subset of the data,

play54:03

and they can get a gradient from that

play54:05

of how to change their parameters,

play54:06

and they can then share all those gradients.

play54:09

So every copy can benefit

play54:11

from what all the other copies extracted from data,

play54:14

and we can't do that.

play54:16

If suppose you had 10,000 people

play54:18

and they went out and they read 10,000 different books,

play54:21

and after they've each read one book,

play54:23

all of them know what's in all the books.

play54:25

We could get to be very smart that way,

play54:27

and that's what these things are doing

play54:29

and so it makes them

play54:30

far superior to us. - And there is education.

play54:31

There's some schooling that we're trying to do that

play54:34

but not in the way. - Yes.

play54:35

But education's just hopeless,

play54:37

I mean hardly worth paying for.

play54:39

(audience laughing)

play54:44

- Except University of Toronto and Stanford.

play54:46

(audience laughing)

play54:48

- I've tried to explain to friends

play54:49

that Geoff has a very sarcastic sense of humor

play54:51

and if you spend enough time around it, you'll get it.

play54:52

But I'll leave it to you to decide

play54:54

whether that was sarcastic.

play54:56

- So the way we exchange knowledge, roughly speaking,

play54:59

this is something of a simplification,

play55:00

but I produce a sentence

play55:02

and you figure out what you have to change in your brain,

play55:04

so you might have said that,

play55:06

that is if you trust me.

play55:10

We can do that with these models too.

play55:13

If you want one neural net architecture

play55:14

to know what another architecture knows,

play55:15

which is a completely different architecture,

play55:16

you can't just give it the weights.

play55:18

So you get one to mimic the output of the other,

play55:20

that's called distillation

play55:22

and that's how we learn from each other.

play55:23

But it's very inefficient,

play55:24

it's limited by the bandwidth of a sentence,

play55:25

which is a few hundred bits.

play55:27

Whereas if you have these models,

play55:28

these digital agents which have a trillion parameters,

play55:32

each of them looks at different bits of data

play55:34

and then they share the gradients,

play55:36

they're sharing a trillion numbers.

play55:38

So you are comparing an ability to share knowledge

play55:41

that's in trillions of numbers

play55:43

with something that's hundreds of bits.

play55:46

They're just much, much better than us at sharing.

play55:49

- So I guess Geoff that-

play55:53

So I agree with you at the technology level,

play55:56

but it sounded like for you

play55:59

that's the moment that got you feeling very negative.

play56:03

- That's the moment I thought, we are history, yeah.

play56:06

- Yeah, I'm less negative than you.

play56:08

I'll explain later,

play56:10

but I think that's where we- - Well let one sec actually,

play56:12

let's talk about that.

play56:14

Explain why you are optimistic

play56:17

and let's understand why you are more pessimistic.

play56:21

- I'm pessimistic 'cause the pessimists are usually right.

play56:24

(audience laughing)

play56:26

- I thought I was a pessimist too.

play56:28

We have this conversation.

play56:30

So I don't know if I should be called an optimist.

play56:34

I think I'm-

play56:37

Look when you came to a country when you're 15

play56:40

now speaking a single bit of language and starting from $0,

play56:46

there's something very pragmatic in my thinking.

play56:50

I think technology,

play56:52

our human relationship with technology is a lot messier

play56:57

than an academia typically would predict,

play57:01

'cause we come to academia in the ivory tower,

play57:04

we want to make a discovery,

play57:06

we want to build a piece of technology,

play57:08

but we tend to be purist.

play57:11

But when the technology like AI hit the ground

play57:14

and reach the societal level,

play57:17

it is inevitably messily entangled with what humans do.

play57:22

And this is where maybe you call it optimism

play57:25

is my sense of humanity.

play57:27

I believe in humanity.

play57:29

I believe in the, not only the resilience of humanity,

play57:33

but also of collective will,

play57:35

the arc of history is dicey sometimes.

play57:39

But if we do the right thing,

play57:44

we have a chance, we have a fighting chance

play57:47

of creating a future that's better.

play57:49

So what I really feel is not

play57:52

delusional optimism at this point,