AI isn't gonna keep improving

Summary

The video script discusses the concept of hitting an 'AI Plateau,' suggesting that despite advancements in AI models like ChatGPT and Claude, we may be nearing a limit in performance improvements due to physical constraints, similar to the end of Moore's Law. It explores the idea that future progress in AI might not come from incremental improvements but from entirely new architectures and approaches. The script also touches on the environmental impact of AI and the need for a shift in focus towards more general AI methods that can scale effectively.

Takeaways

• 🚀 The AI industry might be reaching a plateau in terms of performance improvements, despite the rapid advancements in AI models like GPT-2 to GPT-4 and the emergence of open-source models.
• 📉 The script references Moore's Law, which predicted the doubling of transistors on a microchip every two years, but acknowledges that this trend has slowed due to physical limitations in manufacturing processes.
• 🏭 The manufacturing complexity for advanced chips is so high that only a few companies like TSMC can produce them, leading to a concentration in the industry and affecting the pace of innovation.
• 📊 A study suggests that the performance growth of processors has flattened, with the rate of improvement decreasing significantly from the past decades.
• 💡 The script discusses the potential of different architectures, like Apple's approach with specialized cores for efficiency and performance, as a way to circumvent the limitations of traditional CPUs.
• 🎯 There's a call for a shift in focus from just increasing compute power to developing more general AI methods that can scale effectively, such as search and learning algorithms.
• 🌐 The environmental impact and cost of AI research, which often requires massive amounts of compute resources, are becoming concerns as the field matures.
• 📈 Despite the plateau in performance, there's potential for AI to grow in new areas, like video generation, that are not yet fully explored and could provide the next wave of advancements.
• 🔍 The script highlights the need for new benchmarks and methods to measure AI progress, as current models struggle with tasks that require general intelligence and adaptability.
• 🚧 The future of AI may lie in hybrid models that combine the strengths of both handcrafted algorithms and AI, moving beyond the limitations of large language models (LLMs).

Q & A

What is the AI Plateau mentioned in the script?

The AI Plateau refers to the perceived slowing down of significant advancements in AI capabilities, despite the continuous release of new models. It suggests that we might be reaching a limit in how much AI can improve within the current paradigm.

What is Moore's Law and how does it relate to the script's discussion on AI?

Moore's Law is the observation that the number of transistors on a microchip doubles approximately every two years, leading to increased performance at a lower cost. The script discusses how this law has seemingly plateaued in recent years, which parallels the perceived plateau in AI improvements.

What is the significance of the flatlining in the performance graph presented in the script?

The flatlining in the performance graph indicates that the rate of improvement in AI performance has slowed down, suggesting that we may be reaching a physical or theoretical limit in how much further AI can advance using current technologies and methods.

What does the script suggest about the future of AI if we continue to rely on increasing compute power?

The script suggests that relying solely on increasing compute power may not yield the same exponential improvements in AI as it has in the past. Instead, it implies that innovative architectural changes and new methods will be necessary to continue AI's advancement.

What is the script's view on the comparison between CPUs and AI models in terms of hitting a plateau?

The script draws a parallel between CPUs, which have seen a slowdown in performance improvements due to physical limitations, and AI models, which seem to be reaching a similar plateau in terms of the improvements seen with each new generation.

What is the role of TSMC in the context of chip manufacturing as discussed in the script?

TSMC (Taiwan Semiconductor Manufacturing Company) is highlighted as one of the few companies capable of manufacturing chips at the smallest, most efficient scales. The script notes that reliance on TSMC by major tech companies signifies a bottleneck in innovation due to the complexity of these manufacturing processes.

What does the script imply about the relationship between the number of transistors and AI performance?

The script implies that historically, an increase in the number of transistors has been a measure of progress in processors and by extension, AI performance. However, it suggests that this measure may no longer be as indicative of progress due to the plateau in AI improvements.

What alternative to Moore's Law is proposed in the script, and how does it compare?

The script mentions an alternative law proposed in a study that attempts to account for the放缓 in performance gains. This alternative law is shown as a flatter line compared to Moore's Law, indicating a slower rate of improvement in recent years.

How does the script discuss the potential of different architectures for AI advancement?

The script suggests that different architectures, such as those being researched by companies like IBM for analog AI chips, could potentially offer new paths for AI advancement that bypass the limitations of current models and hardware.

What does the script suggest about the future of AI research and development?

The script suggests that the future of AI research and development may need to focus on creating and iterating on entirely new architectures and methods, rather than simply increasing the compute power or refining existing models.

What is the significance of the ARC AI prize mentioned in the script?

The ARC AI prize is significant as it represents a public competition aimed at encouraging new ideas and solutions in AI that can efficiently acquire new skills and solve open-ended problems, which the script argues is a more accurate representation of general intelligence.

Outlines

00:00

🚀 AI Development and the End of Moore's Law

The paragraph discusses the current state of AI development and draws a parallel to Moore's Law, which predicted the doubling of transistors on a microchip every two years. It highlights the rapid advancements in AI models like GPT-2 to GPT-4 and the emergence of Claude and open-source models like llama and mistol. The speaker then delves into the concept of Moore's Law, its historical significance, and how it led to significant performance improvements in computers. However, the paragraph notes that we've hit a wall in physics, making it difficult to continue this pace of improvement. The speaker points out that companies like TSMC are now the only ones capable of manufacturing chips at the scale required for the most efficient processors, and even they are not meeting the goals set by Moore's Law. The paragraph concludes with a discussion of how the performance gains in AI and computing have started to plateau, similar to the trend seen in Moore's Law.

05:00

🔍 The Plateau in AI and CPU Performance

This paragraph explores the idea that AI and CPU performance improvements may be reaching a plateau. It discusses how companies like Nvidia admit to a slowdown in CPU performance gains, moving from significant yearly improvements to much smaller ones. The speaker uses their own experience with computer hardware to illustrate that the performance difference between a 2020 processor and a 2024 one is not as vast as historical improvements. The paragraph also touches on the benefits of this plateau, such as being able to purchase older processors at a lower cost without a significant loss in performance. It then transitions into a discussion about the potential for new architectures to break through the current limitations, mentioning IBM's research into analog AI chips as an example. The speaker also reflects on the release of new AI models like 'mol large 2' and the thoughts of AI researchers on the future of AI development, suggesting that the focus should shift from incremental improvements to entirely new architectures.

10:02

🌐 The Future of AI and Compute

The paragraph delves into the future of AI and the limitations of traditional CPU manufacturing, suggesting that a new architecture is necessary for further improvements. It discusses the potential of GPUs for AI tasks and contrasts it with the exploration of new AI chip architectures by companies like IBM. The speaker also mentions the release of 'mol large 2' and its capabilities, then compares the development of AI models to the evolution of CPUs, noting a similar trend of diminishing returns on performance improvements. The paragraph includes a tweet from Yan LeCun, emphasizing the need to look beyond large language models (LLMs) for the next generation of AI systems. It concludes with a discussion of Apple's approach to innovation in processor design and the potential for AI to follow a similar path, focusing on specialized tasks and efficiency rather than general performance gains.

15:03

📈 The Diminishing Returns of Compute in AI

This paragraph discusses the diminishing returns on compute power in AI, referencing a 2019 blog post by AI pioneer Rich Sutton. It contrasts the historical approach of encoding expert knowledge into AI systems with the more general methods that have proven more effective. The speaker points out that as we reach the limits of traditional CPU architecture, the focus should shift to more general AI methods that can scale, such as search and learning. The paragraph also touches on the environmental impact of AI's increasing compute demands and the concentration of AI research in large corporations that can afford the necessary resources. It concludes with a call for a change in approach, suggesting that the future of AI lies in finding new methods and architectures rather than simply increasing compute power.

20:05

🚧 The Stalled Progress of Artificial General Intelligence (AGI)

The final paragraph addresses the stagnation in progress towards Artificial General Intelligence (AGI) and the misconceptions surrounding it. It critiques the current benchmarks used to measure AI progress, arguing that they do not accurately reflect general intelligence. The speaker agrees with the definition of AGI as a system capable of efficiently acquiring new skills and solving open-ended problems, rather than just automating economically valuable work. The paragraph highlights the need for new ideas and benchmarks to measure true AGI progress, pointing to initiatives like the Arc Prize, which challenges the AI community to develop solutions that can beat open-source benchmarks. It concludes with a reflection on the plateau in AI development and the need to move beyond current models like LLMs to achieve further advancements in AI capabilities.

Keywords

💡AI Plateau

The term 'AI Plateau' refers to a perceived slowdown in the rapid advancements of artificial intelligence capabilities. In the video, the speaker discusses the possibility that AI development might be reaching a point where significant improvements are becoming less frequent or less impactful. This concept is central to the video's theme, as it explores whether the pace of AI innovation is slowing down, despite the continuous introduction of new models and technologies.

💡Moore's Law

Moore's Law is the observation that the number of transistors on a microchip doubles approximately every two years, which historically led to a corresponding increase in computational power. In the video, the speaker uses Moore's Law as an analogy to discuss the potential slowing down of AI progress. The script mentions how Moore's Law has faced physical limitations, which parallels the discussion on AI reaching a plateau.

💡Transistors

Transistors are semiconductor devices used to amplify or switch electronic signals and electrical power. They are a fundamental component of modern electronic devices. The video script discusses how the doubling of transistors on microchips, as per Moore's Law, has historically driven performance improvements in computers. The speaker reflects on the current state of transistor technology in relation to AI's potential plateau.

💡Manufacturing Processes

Manufacturing processes refer to the methods and techniques used to produce goods or components on a large scale. In the context of the video, the speaker mentions that the complexity of modern chip manufacturing processes has become a limiting factor in the continued advancement of Moore's Law. This ties into the broader discussion about the physical and practical barriers that could also be affecting AI development.

💡Specialized Chips

Specialized chips are designed for specific tasks, as opposed to general-purpose processors. The video script points out that companies like Apple have embedded specialized chips in their processors to optimize certain functions, such as video encoding. This concept is used to draw a parallel with AI, suggesting that specialized architectures might be necessary to overcome the plateau in AI performance.

💡LLMs (Large Language Models)

Large Language Models (LLMs) are AI models trained on vast amounts of text data to understand and generate human-like text. The video discusses how LLMs might be reaching their limits in terms of improvements, which is a central concern as it questions the future of AI progress. The script mentions models like GPT-2, GPT-3, and Claude, indicating a potential plateau in the performance gains of these models.

💡General AI Methods

General AI methods refer to approaches that aim to create AI systems capable of handling a wide range of tasks without needing specific programming for each one. The video script suggests that focusing on general AI methods, rather than trying to encode expert knowledge into AI systems, might be a more fruitful path for AI development. This aligns with the video's exploration of how to overcome the AI plateau.

💡Hype Cycle

A Hype Cycle is a graphical representation of the maturity and adoption of technologies, which typically includes a period of overenthusiasm followed by a period of disillusionment. The video uses the concept of the Hype Cycle to discuss the current state of AI, suggesting that AI might be entering a 'trough of disillusionment' where the reality of AI capabilities does not match the initial hype.

💡AGI (Artificial General Intelligence)

Artificial General Intelligence refers to AI systems that possess the ability to understand, learn, and apply knowledge across a broad range of tasks at a human level. The video script contrasts AGI with the current state of AI, which is more narrowly focused on specific tasks. The speaker discusses the challenges in achieving AGI and how the current trajectory of AI development might not lead to it.

💡Neuromorphic Computing

Neuromorphic computing is an approach to designing computer systems that模仿 the human brain's structure and function. The video script briefly touches on this concept as a potential alternative to traditional computing architectures. It suggests that neuromorphic computing could be a way to overcome the limitations faced by current AI models and hardware.

Highlights

The discussion suggests we might be reaching an AI Plateau, despite advancements in AI models like Chat GPT and Claude.

Mo's Law, which predicted the doubling of transistors on microchips every 2 years, is no longer holding true due to physical limitations.

The manufacturing complexity of microchips has increased, with only a few companies like TSMC capable of producing the smallest, most efficient chips.

A study shows a flatlining of performance improvements in recent years, contrary to the steady growth seen from the 70s to the 2000s.

Nvidia's own data indicates a significant plateau in CPU performance improvements.

The value of older processors has increased as their performance remains competitive with newer, more expensive models.

The discussion points out that GPUs, while not the best for AI tasks, can still see performance improvements by adding more chips to their architecture.

IBM is researching analog AI chips, potentially offering a new architecture for AI that could surpass the limitations of traditional CPUs and GPUs.

Mol, an open-source AI company, released Mol Large 2, a model that significantly improves on code generation, mathematics, and reasoning.

The head of AI at Meta suggests that students interested in next-gen AI systems should look beyond Large Language Models (LLMs).

Apple's strategy of integrating specialized chips for tasks like video encoding shows a path forward for performance improvements.

The discussion predicts a future where AI models become commodities, with performance differences between models diminishing.

The transcript highlights the need for new AI methods that can scale, such as search and learning, rather than trying to encode human knowledge into AI systems.

The fast inverse square root algorithm used in Doom is cited as an example of how clever hacks, rather than raw compute power, can lead to significant advancements.

The Gartner Hype Cycle for AI shows that AI is currently in the 'trough of disillusionment', indicating that expectations may be outpacing realistic progress.

The discussion concludes that the future of AI may not lie in making existing models faster, but in developing entirely new types of AI architectures.

The Arc prize is introduced as a competition to encourage new ideas in AI, highlighting the need for AI to efficiently acquire new skills, not just demonstrate existing ones.

Transcripts

I got a Hut take for y'all today I think

we might have hit an AI Plateau well we

haven't hit it yet I think we're getting

there fast what do I mean this can't be

possible right when we look at all the

new models and all the crazy things you

can do with them the improvements from

chat GPT 2 to 3 to 3.5 to four with

Claude coming out of nowhere and being

really good with open source models like

llama and mistol we can't be out a

plateau that's crazy right well there's

a lot of things that have these patterns

and I want to start with with a bit of

an interesting tangent I want to talk

about Mo's law if you're not familiar

Mo's law is an old concept from the

programming world it's a law created

it's not a real law it was some

speculation from a Dev and a cardware

Enthusiast back in the like what' 7s and

he noticed how fast things were

improving in terms of performance his

observation is that the number of

transistors on a microchip roughly

doubles every 2 years and the cost is

haved over the same time time frame so

if you had a chip let's say that had

four transistors on it within 2 years

with advancements on how we were

manufacturing these chips we' get it up

to eight transistors and it would be

cheaper and we did that over and over

again and saw massive growth in

performance of our machines it was

actually realistic for a bit if you took

a computer that you went and bought at

Best Buy and then waited 2 to 4 years

bought a new one the processor could be

two times faster in a very short window

and it's crazy to think because now like

if you buy an Apple M1 computer from

2020 and a brand new top of the line

machine from 2024 the performance

difference between those things is not

that big but back in the day we saw

insane improvements year over year we've

started to hit walls with the physics

though we realized you can only get so

small for the Silicon before you start

running into manufacturing problems now

the manufacturing processes for a lot of

these things are so complex there's only

one or two companies that can even do it

at the small size that we're expected to

hit now if you want to make the most

efficient chips possible that fit as

many transistors into your die as

possible you have to do that through a

company like tsmc because they're one of

the only places in the world that can

manufacture that way and companies like

Intel apple and Nvidia all rely on that

one manufacturer tsmc that is still not

hitting the mors law goals but they're

the only ones even vaguely close we have

effectively accepted that Mo's law due

to physics is no longer true here we see

from a study where somebody proposed a

new alternative to Mor's law the blue

line is Mora's law the orange line is

their alternative law but the green is

performance and you'll notice things are

flatlining pretty hard up here when

before they were going up at a

relatively steady rate from the' 70s to

the 2000s even like the 2015 things were

going up pretty steadily but we've

started to see a flatline and the harsh

reality is that from 2020 onwards it's

gotten worse not better that's

terrifying obviously there are companies

that disagree here is a diagram from

Nvidia where they actually admit that

the CPU performance we're seeing has

plateaued pretty hard where we went from

getting multiple giant wins to around

1.5x per year down to like 1.1x per year

I'm streaming right now using a PC with

Hardware from when did the 10700k

release date yeah the processor of my

desktops from 2020 it's not even one of

the high-end ones and it doesn't perform

much worse than the top spec one I just

bought a new computer in different room

performance wins year-over-year have

gotten way worse even though the

technology is still advancing we're

still making big wins in the

manufacturing Intel and AMD are as

competitive as ever we're still not

seeing massive wins anymore this does

have benefits though like if you buy an

old processor for way cheaper you still

get really good performance you can buy

a MacBook Air M1 from like Walmart used

for

$400 and have great performance on a

machine that I paid 2 Grand for not long

ago and those things are great I know

people are pitching all about this isn't

about performance as transistor count

we've used the number of transistors as

a way to measure the progress of

processors and historically if you had a

big win in the manufacturing process if

you made the dies go from 10 nanometer

transistors down to like 4 nanometers

you would see massive winds this is

rough obviously nvidia's doing their

thing here claiming that GPU compute

performance continue to grow the fun

thing with graphics cards is they don't

have the same model with cores with the

complexity of sharing things between

them because the cores in a GPU are

significantly Dumber it's a different

abstraction which means you can just

staple more and more gpus onto each

other to improve performance you might

end up with a GPU no longer being a tiny

little thing you slide into your

computer and now it's a giant room full

of things it's still one GPU because of

the way the chips are architected but

the only way Nvidia is going to see this

type of performance wins continuously is

if they just add more and more chips to

their actual architecture it's kind of

cheating but the reality is that the

tech that we use today which is

traditional CPU manufacturing we have

hit a physics wall for how much

improvement we can see and the only way

to get out of it theoretically is an

entirely different architecture and way

of building compute things that rely on

this model will not benefit from these

advancements as much but anything that

can work with this model could

theoretically continue to see growth on

that note gpus are not necessarily the

best way to do AI stuff just a quick tie

in I think it's interesting that IBM is

researching analog AI chips similar to

the stuff that we saw with Bitcoin back

in the day where before you would mine

Bitcoin with a GPU before as6 were made

which were specialized computers just to

make Bitcoin mining as efficient as

possible we're starting to see some

research into doing this for AI as well

which is exciting potentially gpus

aren't the right architecture for AI and

we can see advancements and these chips

once they work will probably Advance

significantly faster than CPUs or gpus

so why am I talking about all of this

when I'm talking about models hell why

am I even talking about models I saw a

very interesting post for mistol mol is

one of the two big open-source AI

businesses it's them and funny enough

meta So Meta faceb are working on llama

which is their open- Source model it's

technically not open source because you

can't run the code yourself but you get

the model and you can use it however you

want mol is doing the same thing and

they just released mol large 2 the new

generation of their Flagship model

compared to pror mist large 2 is

significantly more capable in code gen

mathematics and reason

it also has stronger multi language

stuff and function calling stuff cool

the key here is large enough this made

me start thinking a lot about the

plateau that we're likely reaching and

I'm not the only one thinking about this

here's a tweet from Yan laon who is the

head of AI and llm research at Facebook

and meta he's one of the ones most

directly responsible for the creation of

llama and he said if you're a student

interested in building the next

generation of AI systems don't work on

llms what llms are how all of these

things work well let's rephrase this if

you're a student interested in building

the next generation of computers don't

work on CPUs or don't work at Intel it's

obvious when you look at the

numbers that iteration on CPUs is not

going to be where we see massive

performance wins and massive computation

wins going forward different

architectures will have to be invented

and iterated on for us to see meaningful

improvements in performance

year-over-year Apple does this in all

sorts of interesting ways one of the

crazy things Apple invented was the idea

of having different cores with different

roles so you had efficiency cores that

are trying to use as little power as

possible to do simple things and then

performance cores that use way more

power but are quite a bit more powerful

they also started embedding things like

video processing and video encoding

chips that just do h264 h265 decoding

and encoding way more efficiently Apple

started adding things to their

processors that weren't just CP and also

weren't just gpus in order to optimize

specific things so they could keep

seeing massive performance wins I think

this is the future for AI as well and I

have a reason I have a very similar

chart to this one notice how much

smaller the winds are getting Claude saw

another solid one with Sonet in 3.5 but

the Gap from gp4 Turbo to Turbo 2 to 40

is a lot smaller than from four turbo to

four it is way smaller than from 4 to

three

CLA one to two to three some massive

winds but those are starting to slow

down as well we're seeing a plateau of

the quality of the responses these

models are generating it is not like

going from 4 to 4 Turbo to 40 was less

work than going from 3.5 to 4 if

anything there is more money more time

more gpus more effort going into these

bumps and the actual bump we're seeing

is going down so each of these ations

takes more money more time more compute

more energy and the results are not as

big as they used to be I know a lot of

people are saying the AI future is going

to Doom us all because the AI keep

getting so much smarter eventually

they're going to be smarter than all of

us I don't see that here I don't see

that here at all what I see is a

theoretical ceiling that we're getting

very close to and a closing of the Gap

in performance between these different

models more and more these options are

going to become Commodities the same way

you have like 15 different computer

manufacturers is making the same Windows

laptop that has roughly the exact same

performance we're starting to see that

here too I have to read a LinkedIn post

which I know pain cringe miserable so

I'm going to soften the blow with an XK

CD first this one was linked in chat and

I thought it was really funny number of

computers created is going up a lot

year-over-year in fact I think it's

going up exponentially but the number

destroyed by hurling them into Jupiter

it's a much smaller number it's only

three so far NASA needs to pick up the

pace if they ever want to finish the job

yeah they ever want to catch up they got

work to do it's a fun way to think about

data in these ways the compute changes

over time anyways the bitter lesson

famous 2019 blog post claims that

General AI methods using massive compute

are the most effective nvidia's soaring

stock price supports the thesis but is

this approach sustainable what are the

Alternatives in the original blog post

AI Pioneer Rich suon makes the following

observations over the last 70 years AI

researchers have repeatedly made the

same mistakes of trying to bake human

knowledge in into AI systems only to be

eventually outperformed by a more

General method using Brute Force compute

this is funny cuz we're seeing the

opposite in processors now where

processors were trying to just increase

how many transistors were in them and

how fast they could solve problems and

now we're seeing specialized chips being

embedded in the processors that do

certain things way better some prominent

examples of what was happening before

with models were custom chess and go

engines versus deep blue and Alpha zero

this was a fun one the go not the

programming language the board game was

really hard for software developers to

solve because the game has so many

different potentials you can't just

encode all of them and then figure out

which is optimal and we learned after

trying to make custom engines for these

things that AI Solutions like deep blue

and Alpha zero that were more generic

more traditional AI did a better job

than the custom code we wrote it took

hilariously more compute to do it like

hundreds of times more but the results

were always better the main reasons for

this are the following building an

expert knowledge is personally

satisfying for the experts and often

useful in the short term it's a very

good point if you have experts that know

this game really well or Know video

encoding really well they can Flex their

knowledge feel useful and see an

immediate result all of which feels good

on top of that researchers tend to think

in terms of fixed availability compute

when it's actually increasing daily this

is also a fair point yes the amount that

a given processor improves

year-over-year has gone down but the

amount of processors you have available

is going up especially with Nvidia going

insane with their manufacturing Sun

concludes that we should focus on on

General AI methods that can continue to

scale most notably search and learning

we should stop trying to bake the

contents of the human mind into AI

systems as they are too complex and

instead focus on finding metame methods

that can capture this complexity

themselves some of the important things

that people pointed out are that mors

law is fading architecture of our most

successful learning models were actually

carefully handcrafted by humans like

Transformers comets lstms Etc and for

General computation problems like

integer factorization progress based on

human understanding was often far

greater than progress according to Mo's

law another great point we're still

optimizing algorithms in ways that we

never would have imagined possible

before one that I love to cite here is

the fast inverse square root which was

used in Doom in order to handle lighting

Reflections and rendering because

knowing the inverse square root lets you

know how far something is relative to

multiple points and it's used a ton for

doing math in games previously getting

this number getting the inverse square

root took a lot of compute and as such

the idea of 3D games was basically

impossible but someone discovered a math

hack they didn't even understand at the

time the fast inverse square root

function that was in this code base had

evil floating Point bit level hacking

it's the comment here this weird bit

shift where they take this random

hardcoded value subtract the bit shifted

long long representation of Y comment

what the next comment first

iteration where we multiply it by three

haves and this function here and we

could run it again if we wanted to be

more accurate 3D Graphics program is

supposed perform millions of these

calculations every second to simulate

lighting when code was developed in the

early 90s most floating Point processing

power lagged the speed of integer

processing so yeah if you were trying to

do this with floating points which

everyone was it would eat your processor

the advantages in speed in this fast

function came from treating the 32-bit

floating Point word as an integer then

subtracting it from a magic constant

this integer subtraction in bit shift

resulted in a bit pattern which when

redefined as a floating Point number is

a rough approximation of the inverse

square root of that number this function

this crazy math hack allowed us to add

Dynamic lighting to 3D games this wasn't

something we got because processors were

way more powerful it was a clever hack

that allowed us to invent a new genre of

game effectively pretty nuts pretty

crazy stuff that this enabled as much as

it enabled because somebody came up with

a clever math hack that's not even that

accurate it's just accurate enough so as

is said here the wins we saw on compute

the revolution in 3D games that we saw

after that code came out and people

started using the engine that wasn't

because gpus or CPUs got way better it's

because our understanding of how to use

them to do these specific things got

better and we saw massive wins not

because the CPU got way faster but

because we found smarter ways to use it

and I think this is going to be true now

more than ever in the same way we're

reaching the cap of how much you can do

with a C CPU we're reaching the cap of

how much you can do with an llm

companies like open AI show that

focusing on more compute may still lead

to massive gains as compute power

despite the warning of Mo's law

continues to increase several orders of

magnitude over the next decades don't

necessarily agree currently the hype is

definitely outperforming Mo's law see

the image below as a result AI is at

risk of creating a deep environmental

footprint and research is increasingly

restricted to large corporations that

can afford to pay for the compute it's a

bitter lesson of the last year yeah this

is a fun one Mo's law versus AI

popularity but again Moors law is

plateauing and AI is now way more

popular than what Mor's law enables so

we're just spending billions on gpus

found a surprisingly good chart from

Gartner believe it or

not the hype cycle for artificial

intelligence hype Cycles are very

common this particular chart the startup

hype cycle an idea happens we have a

spike of excitement first Valley of

Death happens where you realize this is

hard you go hard you go really

hard you get inflated expectations

irrational exuberance and then pain you

end up in this thing called the trough

of disillusionment where you're unsure

of everything then the slow slope of

reality as you figure out what you're

actually capable of and what your

product company Vision whatever it is

actually could resolve to and then you

hit the real company and real value so

back to the Gartner chart it's funny

they have all these examples in here

first principles AI multi M agent

systems neuros symbolic AI more and more

things happening and we start getting

into generative Ai and then we hit a

massive Point realized we needed more

optimization things like synthetic data

better model optimization AI that is on

the edge so to speak so it runs on our

phones instead of on the servers

knowledge graphs but you notice we're

going down because these things aren't

fun these things suck and they're

necessary for us to keep evolving then

we started seeing AI makers and teaching

kits to try and get people to actually

learn autonomous vehicles with

which were very painful and still are

cars that drive themselves are far from

functioning but now we're seeing more

and more things that will hopefully

allow us to really benefit from AI but

we need to make sure our expectations

are realistically set not around the

exponential growth every year rather

around how we apply the functionality of

these things to actually benefit Our

Lives day in and day out I am honestly

just annoyed that people pretend the

models are going to get two times better

every couple years because we went

through that that's clearly over we're

just not seeing levels up like that

anymore what I'm expecting us to see

instead is massive winds in things that

we're not currently using models 4 like

we're starting to see video generation

catch on and it's taking us a lot of

time to get there but I could see us

growing there really quickly similar to

how chat GPT got way better really

quickly but it will also hit a plateau

and I think we're going to see more and

more of those plateaus hit and our

solution isn't going to be magically

make it better it's going to be entirely

different models and hybrids where we

take advantage of handwritten and

crafted code maybe human massaging of

things and AIS and intermingling and

mixing those the same way CPUs and gpus

take turns working on things depending

on what each is best at handwritten code

and AI code doing similar stuff has a

ton of potential and I think that's

going to be the future of AI because

this this is not the future of AI this

is a flatline this is a plateau This Is

Us reaching the end not the beginning

and if mistol is saying that their

model's large enough I'm inclined to

agree especially when you look at the

numbers here and you see how close all

of these models are getting to being

basically even the winds are no longer

the models being way better than the

others the winds are going to be

efficiency performance speed of

responses and then the next set of wins

is going to be how we use these things

in new and unique ways this is actually

a very interesting link there's a

project called The Arc prize that was

just linked from chat AGI progress has

stalled new ideas are needed

it's a million dooll public competition

to beat an open- Source a solution to

the ark AGI

Benchmark most AI benchmarks measure

skill but skill is not intelligence

general intelligence is the ability to

efficiently acquire new skills charlot's

unbeaten 2019 abstraction and reasoning

Corpus for artificial general

intelligence is the only formal

Benchmark of AGI it's easy for humans

but it's hard for

AI oh this is fun this is this is going

to be like captions basically so we have

these patterns an input and an output

it's pretty clear what we do here

configure your output grid there and

then we have to put the dark Blues here

here here and

here submit fun so the point here is

these are the types of puzzles that we

can Intuit it we look at the pattern and

we can learn quickly what the pattern is

with these things it looks like the

light blue is ignored

red has the outward pattern then dark

blue has the pattern with the like

t-shape but AI is historically really

bad at solving these types of things so

here's the arc AGI progress but if we

look at other AI benchmarks that people

use a lot of the ones we were looking at

earlier like H swag imag net all of

these it seems like things are improving

at an insane rate when you look at

general intelligence through a benchmark

like this AI sucks at it progress

towards artificial general intelligence

is stalled llms are trained on

unimaginably vast amounts of data yet

they remain unable to adapt to simple

problems they haven't been trained on or

make novel inventions no matter how

basic strong Market incentives have

pushed Frontier AI research to go closed

Source research attention and resources

are being put towards a dead end you can

change that I like that they have a they

call out that the consensus definition

for AGI is wrong AGI is a system that

can automate the majority of

economically valuable work but in

reality AGI is a system that can

efficiently acquire new skills and solve

open-ended problems yes that's what the

General in AGI stands for I actually

fully agree with this call out defition

are important because we turn them into

benchmarks to measure progress towards

AI I fully agree I love that Nat fredman

is one of the advisers the old CEO of

GitHub we also have Mike knop who's an

absolute Legend who's been involved in

all things software Dev and AI for a

very long time yeah I love this and I

think this is the only way we're going

to really see improvements and wins with

AI llms are hitting their limitations

and as we saw here they're not really

winning on General benchmarks like this

and sure we have these fancy benchmarks

that everybody loves but even these

we're starting to see a flatline and a

plateau on them we might be at the end

of the llm Revolution and if we want to

see AI continue to grow and advance in

its capabilities we might have to leave

behind llms the same way we're starting

to leave behind CPUs the future isn't an

llm but faster if we want the future to

be AI it has to be a different type of

AI let me know what you guys think and

tell me all the ways I'm wrong until

next time peace nerds