AI isn't gonna keep improving
Summary
TLDRThe video script discusses the concept of hitting an 'AI Plateau,' suggesting that despite advancements in AI models like ChatGPT and Claude, we may be nearing a limit in performance improvements due to physical constraints, similar to the end of Moore's Law. It explores the idea that future progress in AI might not come from incremental improvements but from entirely new architectures and approaches. The script also touches on the environmental impact of AI and the need for a shift in focus towards more general AI methods that can scale effectively.
Takeaways
- 🚀 The AI industry might be reaching a plateau in terms of performance improvements, despite the rapid advancements in AI models like GPT-2 to GPT-4 and the emergence of open-source models.
- 📉 The script references Moore's Law, which predicted the doubling of transistors on a microchip every two years, but acknowledges that this trend has slowed due to physical limitations in manufacturing processes.
- 🏭 The manufacturing complexity for advanced chips is so high that only a few companies like TSMC can produce them, leading to a concentration in the industry and affecting the pace of innovation.
- 📊 A study suggests that the performance growth of processors has flattened, with the rate of improvement decreasing significantly from the past decades.
- 💡 The script discusses the potential of different architectures, like Apple's approach with specialized cores for efficiency and performance, as a way to circumvent the limitations of traditional CPUs.
- 🎯 There's a call for a shift in focus from just increasing compute power to developing more general AI methods that can scale effectively, such as search and learning algorithms.
- 🌐 The environmental impact and cost of AI research, which often requires massive amounts of compute resources, are becoming concerns as the field matures.
- 📈 Despite the plateau in performance, there's potential for AI to grow in new areas, like video generation, that are not yet fully explored and could provide the next wave of advancements.
- 🔍 The script highlights the need for new benchmarks and methods to measure AI progress, as current models struggle with tasks that require general intelligence and adaptability.
- 🚧 The future of AI may lie in hybrid models that combine the strengths of both handcrafted algorithms and AI, moving beyond the limitations of large language models (LLMs).
Q & A
What is the AI Plateau mentioned in the script?
-The AI Plateau refers to the perceived slowing down of significant advancements in AI capabilities, despite the continuous release of new models. It suggests that we might be reaching a limit in how much AI can improve within the current paradigm.
What is Moore's Law and how does it relate to the script's discussion on AI?
-Moore's Law is the observation that the number of transistors on a microchip doubles approximately every two years, leading to increased performance at a lower cost. The script discusses how this law has seemingly plateaued in recent years, which parallels the perceived plateau in AI improvements.
What is the significance of the flatlining in the performance graph presented in the script?
-The flatlining in the performance graph indicates that the rate of improvement in AI performance has slowed down, suggesting that we may be reaching a physical or theoretical limit in how much further AI can advance using current technologies and methods.
What does the script suggest about the future of AI if we continue to rely on increasing compute power?
-The script suggests that relying solely on increasing compute power may not yield the same exponential improvements in AI as it has in the past. Instead, it implies that innovative architectural changes and new methods will be necessary to continue AI's advancement.
What is the script's view on the comparison between CPUs and AI models in terms of hitting a plateau?
-The script draws a parallel between CPUs, which have seen a slowdown in performance improvements due to physical limitations, and AI models, which seem to be reaching a similar plateau in terms of the improvements seen with each new generation.
What is the role of TSMC in the context of chip manufacturing as discussed in the script?
-TSMC (Taiwan Semiconductor Manufacturing Company) is highlighted as one of the few companies capable of manufacturing chips at the smallest, most efficient scales. The script notes that reliance on TSMC by major tech companies signifies a bottleneck in innovation due to the complexity of these manufacturing processes.
What does the script imply about the relationship between the number of transistors and AI performance?
-The script implies that historically, an increase in the number of transistors has been a measure of progress in processors and by extension, AI performance. However, it suggests that this measure may no longer be as indicative of progress due to the plateau in AI improvements.
What alternative to Moore's Law is proposed in the script, and how does it compare?
-The script mentions an alternative law proposed in a study that attempts to account for the放缓 in performance gains. This alternative law is shown as a flatter line compared to Moore's Law, indicating a slower rate of improvement in recent years.
How does the script discuss the potential of different architectures for AI advancement?
-The script suggests that different architectures, such as those being researched by companies like IBM for analog AI chips, could potentially offer new paths for AI advancement that bypass the limitations of current models and hardware.
What does the script suggest about the future of AI research and development?
-The script suggests that the future of AI research and development may need to focus on creating and iterating on entirely new architectures and methods, rather than simply increasing the compute power or refining existing models.
What is the significance of the ARC AI prize mentioned in the script?
-The ARC AI prize is significant as it represents a public competition aimed at encouraging new ideas and solutions in AI that can efficiently acquire new skills and solve open-ended problems, which the script argues is a more accurate representation of general intelligence.
Outlines
🚀 AI Development and the End of Moore's Law
The paragraph discusses the current state of AI development and draws a parallel to Moore's Law, which predicted the doubling of transistors on a microchip every two years. It highlights the rapid advancements in AI models like GPT-2 to GPT-4 and the emergence of Claude and open-source models like llama and mistol. The speaker then delves into the concept of Moore's Law, its historical significance, and how it led to significant performance improvements in computers. However, the paragraph notes that we've hit a wall in physics, making it difficult to continue this pace of improvement. The speaker points out that companies like TSMC are now the only ones capable of manufacturing chips at the scale required for the most efficient processors, and even they are not meeting the goals set by Moore's Law. The paragraph concludes with a discussion of how the performance gains in AI and computing have started to plateau, similar to the trend seen in Moore's Law.
🔍 The Plateau in AI and CPU Performance
This paragraph explores the idea that AI and CPU performance improvements may be reaching a plateau. It discusses how companies like Nvidia admit to a slowdown in CPU performance gains, moving from significant yearly improvements to much smaller ones. The speaker uses their own experience with computer hardware to illustrate that the performance difference between a 2020 processor and a 2024 one is not as vast as historical improvements. The paragraph also touches on the benefits of this plateau, such as being able to purchase older processors at a lower cost without a significant loss in performance. It then transitions into a discussion about the potential for new architectures to break through the current limitations, mentioning IBM's research into analog AI chips as an example. The speaker also reflects on the release of new AI models like 'mol large 2' and the thoughts of AI researchers on the future of AI development, suggesting that the focus should shift from incremental improvements to entirely new architectures.
🌐 The Future of AI and Compute
The paragraph delves into the future of AI and the limitations of traditional CPU manufacturing, suggesting that a new architecture is necessary for further improvements. It discusses the potential of GPUs for AI tasks and contrasts it with the exploration of new AI chip architectures by companies like IBM. The speaker also mentions the release of 'mol large 2' and its capabilities, then compares the development of AI models to the evolution of CPUs, noting a similar trend of diminishing returns on performance improvements. The paragraph includes a tweet from Yan LeCun, emphasizing the need to look beyond large language models (LLMs) for the next generation of AI systems. It concludes with a discussion of Apple's approach to innovation in processor design and the potential for AI to follow a similar path, focusing on specialized tasks and efficiency rather than general performance gains.
📈 The Diminishing Returns of Compute in AI
This paragraph discusses the diminishing returns on compute power in AI, referencing a 2019 blog post by AI pioneer Rich Sutton. It contrasts the historical approach of encoding expert knowledge into AI systems with the more general methods that have proven more effective. The speaker points out that as we reach the limits of traditional CPU architecture, the focus should shift to more general AI methods that can scale, such as search and learning. The paragraph also touches on the environmental impact of AI's increasing compute demands and the concentration of AI research in large corporations that can afford the necessary resources. It concludes with a call for a change in approach, suggesting that the future of AI lies in finding new methods and architectures rather than simply increasing compute power.
🚧 The Stalled Progress of Artificial General Intelligence (AGI)
The final paragraph addresses the stagnation in progress towards Artificial General Intelligence (AGI) and the misconceptions surrounding it. It critiques the current benchmarks used to measure AI progress, arguing that they do not accurately reflect general intelligence. The speaker agrees with the definition of AGI as a system capable of efficiently acquiring new skills and solving open-ended problems, rather than just automating economically valuable work. The paragraph highlights the need for new ideas and benchmarks to measure true AGI progress, pointing to initiatives like the Arc Prize, which challenges the AI community to develop solutions that can beat open-source benchmarks. It concludes with a reflection on the plateau in AI development and the need to move beyond current models like LLMs to achieve further advancements in AI capabilities.
Mindmap
Keywords
💡AI Plateau
💡Moore's Law
💡Transistors
💡Manufacturing Processes
💡Specialized Chips
💡LLMs (Large Language Models)
💡General AI Methods
💡Hype Cycle
💡AGI (Artificial General Intelligence)
💡Neuromorphic Computing
Highlights
The discussion suggests we might be reaching an AI Plateau, despite advancements in AI models like Chat GPT and Claude.
Mo's Law, which predicted the doubling of transistors on microchips every 2 years, is no longer holding true due to physical limitations.
The manufacturing complexity of microchips has increased, with only a few companies like TSMC capable of producing the smallest, most efficient chips.
A study shows a flatlining of performance improvements in recent years, contrary to the steady growth seen from the 70s to the 2000s.
Nvidia's own data indicates a significant plateau in CPU performance improvements.
The value of older processors has increased as their performance remains competitive with newer, more expensive models.
The discussion points out that GPUs, while not the best for AI tasks, can still see performance improvements by adding more chips to their architecture.
IBM is researching analog AI chips, potentially offering a new architecture for AI that could surpass the limitations of traditional CPUs and GPUs.
Mol, an open-source AI company, released Mol Large 2, a model that significantly improves on code generation, mathematics, and reasoning.
The head of AI at Meta suggests that students interested in next-gen AI systems should look beyond Large Language Models (LLMs).
Apple's strategy of integrating specialized chips for tasks like video encoding shows a path forward for performance improvements.
The discussion predicts a future where AI models become commodities, with performance differences between models diminishing.
The transcript highlights the need for new AI methods that can scale, such as search and learning, rather than trying to encode human knowledge into AI systems.
The fast inverse square root algorithm used in Doom is cited as an example of how clever hacks, rather than raw compute power, can lead to significant advancements.
The Gartner Hype Cycle for AI shows that AI is currently in the 'trough of disillusionment', indicating that expectations may be outpacing realistic progress.
The discussion concludes that the future of AI may not lie in making existing models faster, but in developing entirely new types of AI architectures.
The Arc prize is introduced as a competition to encourage new ideas in AI, highlighting the need for AI to efficiently acquire new skills, not just demonstrate existing ones.
Transcripts
I got a Hut take for y'all today I think
we might have hit an AI Plateau well we
haven't hit it yet I think we're getting
there fast what do I mean this can't be
possible right when we look at all the
new models and all the crazy things you
can do with them the improvements from
chat GPT 2 to 3 to 3.5 to four with
Claude coming out of nowhere and being
really good with open source models like
llama and mistol we can't be out a
plateau that's crazy right well there's
a lot of things that have these patterns
and I want to start with with a bit of
an interesting tangent I want to talk
about Mo's law if you're not familiar
Mo's law is an old concept from the
programming world it's a law created
it's not a real law it was some
speculation from a Dev and a cardware
Enthusiast back in the like what' 7s and
he noticed how fast things were
improving in terms of performance his
observation is that the number of
transistors on a microchip roughly
doubles every 2 years and the cost is
haved over the same time time frame so
if you had a chip let's say that had
four transistors on it within 2 years
with advancements on how we were
manufacturing these chips we' get it up
to eight transistors and it would be
cheaper and we did that over and over
again and saw massive growth in
performance of our machines it was
actually realistic for a bit if you took
a computer that you went and bought at
Best Buy and then waited 2 to 4 years
bought a new one the processor could be
two times faster in a very short window
and it's crazy to think because now like
if you buy an Apple M1 computer from
2020 and a brand new top of the line
machine from 2024 the performance
difference between those things is not
that big but back in the day we saw
insane improvements year over year we've
started to hit walls with the physics
though we realized you can only get so
small for the Silicon before you start
running into manufacturing problems now
the manufacturing processes for a lot of
these things are so complex there's only
one or two companies that can even do it
at the small size that we're expected to
hit now if you want to make the most
efficient chips possible that fit as
many transistors into your die as
possible you have to do that through a
company like tsmc because they're one of
the only places in the world that can
manufacture that way and companies like
Intel apple and Nvidia all rely on that
one manufacturer tsmc that is still not
hitting the mors law goals but they're
the only ones even vaguely close we have
effectively accepted that Mo's law due
to physics is no longer true here we see
from a study where somebody proposed a
new alternative to Mor's law the blue
line is Mora's law the orange line is
their alternative law but the green is
performance and you'll notice things are
flatlining pretty hard up here when
before they were going up at a
relatively steady rate from the' 70s to
the 2000s even like the 2015 things were
going up pretty steadily but we've
started to see a flatline and the harsh
reality is that from 2020 onwards it's
gotten worse not better that's
terrifying obviously there are companies
that disagree here is a diagram from
Nvidia where they actually admit that
the CPU performance we're seeing has
plateaued pretty hard where we went from
getting multiple giant wins to around
1.5x per year down to like 1.1x per year
I'm streaming right now using a PC with
Hardware from when did the 10700k
release date yeah the processor of my
desktops from 2020 it's not even one of
the high-end ones and it doesn't perform
much worse than the top spec one I just
bought a new computer in different room
performance wins year-over-year have
gotten way worse even though the
technology is still advancing we're
still making big wins in the
manufacturing Intel and AMD are as
competitive as ever we're still not
seeing massive wins anymore this does
have benefits though like if you buy an
old processor for way cheaper you still
get really good performance you can buy
a MacBook Air M1 from like Walmart used
for
$400 and have great performance on a
machine that I paid 2 Grand for not long
ago and those things are great I know
people are pitching all about this isn't
about performance as transistor count
we've used the number of transistors as
a way to measure the progress of
processors and historically if you had a
big win in the manufacturing process if
you made the dies go from 10 nanometer
transistors down to like 4 nanometers
you would see massive winds this is
rough obviously nvidia's doing their
thing here claiming that GPU compute
performance continue to grow the fun
thing with graphics cards is they don't
have the same model with cores with the
complexity of sharing things between
them because the cores in a GPU are
significantly Dumber it's a different
abstraction which means you can just
staple more and more gpus onto each
other to improve performance you might
end up with a GPU no longer being a tiny
little thing you slide into your
computer and now it's a giant room full
of things it's still one GPU because of
the way the chips are architected but
the only way Nvidia is going to see this
type of performance wins continuously is
if they just add more and more chips to
their actual architecture it's kind of
cheating but the reality is that the
tech that we use today which is
traditional CPU manufacturing we have
hit a physics wall for how much
improvement we can see and the only way
to get out of it theoretically is an
entirely different architecture and way
of building compute things that rely on
this model will not benefit from these
advancements as much but anything that
can work with this model could
theoretically continue to see growth on
that note gpus are not necessarily the
best way to do AI stuff just a quick tie
in I think it's interesting that IBM is
researching analog AI chips similar to
the stuff that we saw with Bitcoin back
in the day where before you would mine
Bitcoin with a GPU before as6 were made
which were specialized computers just to
make Bitcoin mining as efficient as
possible we're starting to see some
research into doing this for AI as well
which is exciting potentially gpus
aren't the right architecture for AI and
we can see advancements and these chips
once they work will probably Advance
significantly faster than CPUs or gpus
so why am I talking about all of this
when I'm talking about models hell why
am I even talking about models I saw a
very interesting post for mistol mol is
one of the two big open-source AI
businesses it's them and funny enough
meta So Meta faceb are working on llama
which is their open- Source model it's
technically not open source because you
can't run the code yourself but you get
the model and you can use it however you
want mol is doing the same thing and
they just released mol large 2 the new
generation of their Flagship model
compared to pror mist large 2 is
significantly more capable in code gen
mathematics and reason
it also has stronger multi language
stuff and function calling stuff cool
the key here is large enough this made
me start thinking a lot about the
plateau that we're likely reaching and
I'm not the only one thinking about this
here's a tweet from Yan laon who is the
head of AI and llm research at Facebook
and meta he's one of the ones most
directly responsible for the creation of
llama and he said if you're a student
interested in building the next
generation of AI systems don't work on
llms what llms are how all of these
things work well let's rephrase this if
you're a student interested in building
the next generation of computers don't
work on CPUs or don't work at Intel it's
obvious when you look at the
numbers that iteration on CPUs is not
going to be where we see massive
performance wins and massive computation
wins going forward different
architectures will have to be invented
and iterated on for us to see meaningful
improvements in performance
year-over-year Apple does this in all
sorts of interesting ways one of the
crazy things Apple invented was the idea
of having different cores with different
roles so you had efficiency cores that
are trying to use as little power as
possible to do simple things and then
performance cores that use way more
power but are quite a bit more powerful
they also started embedding things like
video processing and video encoding
chips that just do h264 h265 decoding
and encoding way more efficiently Apple
started adding things to their
processors that weren't just CP and also
weren't just gpus in order to optimize
specific things so they could keep
seeing massive performance wins I think
this is the future for AI as well and I
have a reason I have a very similar
chart to this one notice how much
smaller the winds are getting Claude saw
another solid one with Sonet in 3.5 but
the Gap from gp4 Turbo to Turbo 2 to 40
is a lot smaller than from four turbo to
four it is way smaller than from 4 to
three
CLA one to two to three some massive
winds but those are starting to slow
down as well we're seeing a plateau of
the quality of the responses these
models are generating it is not like
going from 4 to 4 Turbo to 40 was less
work than going from 3.5 to 4 if
anything there is more money more time
more gpus more effort going into these
bumps and the actual bump we're seeing
is going down so each of these ations
takes more money more time more compute
more energy and the results are not as
big as they used to be I know a lot of
people are saying the AI future is going
to Doom us all because the AI keep
getting so much smarter eventually
they're going to be smarter than all of
us I don't see that here I don't see
that here at all what I see is a
theoretical ceiling that we're getting
very close to and a closing of the Gap
in performance between these different
models more and more these options are
going to become Commodities the same way
you have like 15 different computer
manufacturers is making the same Windows
laptop that has roughly the exact same
performance we're starting to see that
here too I have to read a LinkedIn post
which I know pain cringe miserable so
I'm going to soften the blow with an XK
CD first this one was linked in chat and
I thought it was really funny number of
computers created is going up a lot
year-over-year in fact I think it's
going up exponentially but the number
destroyed by hurling them into Jupiter
it's a much smaller number it's only
three so far NASA needs to pick up the
pace if they ever want to finish the job
yeah they ever want to catch up they got
work to do it's a fun way to think about
data in these ways the compute changes
over time anyways the bitter lesson
famous 2019 blog post claims that
General AI methods using massive compute
are the most effective nvidia's soaring
stock price supports the thesis but is
this approach sustainable what are the
Alternatives in the original blog post
AI Pioneer Rich suon makes the following
observations over the last 70 years AI
researchers have repeatedly made the
same mistakes of trying to bake human
knowledge in into AI systems only to be
eventually outperformed by a more
General method using Brute Force compute
this is funny cuz we're seeing the
opposite in processors now where
processors were trying to just increase
how many transistors were in them and
how fast they could solve problems and
now we're seeing specialized chips being
embedded in the processors that do
certain things way better some prominent
examples of what was happening before
with models were custom chess and go
engines versus deep blue and Alpha zero
this was a fun one the go not the
programming language the board game was
really hard for software developers to
solve because the game has so many
different potentials you can't just
encode all of them and then figure out
which is optimal and we learned after
trying to make custom engines for these
things that AI Solutions like deep blue
and Alpha zero that were more generic
more traditional AI did a better job
than the custom code we wrote it took
hilariously more compute to do it like
hundreds of times more but the results
were always better the main reasons for
this are the following building an
expert knowledge is personally
satisfying for the experts and often
useful in the short term it's a very
good point if you have experts that know
this game really well or Know video
encoding really well they can Flex their
knowledge feel useful and see an
immediate result all of which feels good
on top of that researchers tend to think
in terms of fixed availability compute
when it's actually increasing daily this
is also a fair point yes the amount that
a given processor improves
year-over-year has gone down but the
amount of processors you have available
is going up especially with Nvidia going
insane with their manufacturing Sun
concludes that we should focus on on
General AI methods that can continue to
scale most notably search and learning
we should stop trying to bake the
contents of the human mind into AI
systems as they are too complex and
instead focus on finding metame methods
that can capture this complexity
themselves some of the important things
that people pointed out are that mors
law is fading architecture of our most
successful learning models were actually
carefully handcrafted by humans like
Transformers comets lstms Etc and for
General computation problems like
integer factorization progress based on
human understanding was often far
greater than progress according to Mo's
law another great point we're still
optimizing algorithms in ways that we
never would have imagined possible
before one that I love to cite here is
the fast inverse square root which was
used in Doom in order to handle lighting
Reflections and rendering because
knowing the inverse square root lets you
know how far something is relative to
multiple points and it's used a ton for
doing math in games previously getting
this number getting the inverse square
root took a lot of compute and as such
the idea of 3D games was basically
impossible but someone discovered a math
hack they didn't even understand at the
time the fast inverse square root
function that was in this code base had
evil floating Point bit level hacking
it's the comment here this weird bit
shift where they take this random
hardcoded value subtract the bit shifted
long long representation of Y comment
what the next comment first
iteration where we multiply it by three
haves and this function here and we
could run it again if we wanted to be
more accurate 3D Graphics program is
supposed perform millions of these
calculations every second to simulate
lighting when code was developed in the
early 90s most floating Point processing
power lagged the speed of integer
processing so yeah if you were trying to
do this with floating points which
everyone was it would eat your processor
the advantages in speed in this fast
function came from treating the 32-bit
floating Point word as an integer then
subtracting it from a magic constant
this integer subtraction in bit shift
resulted in a bit pattern which when
redefined as a floating Point number is
a rough approximation of the inverse
square root of that number this function
this crazy math hack allowed us to add
Dynamic lighting to 3D games this wasn't
something we got because processors were
way more powerful it was a clever hack
that allowed us to invent a new genre of
game effectively pretty nuts pretty
crazy stuff that this enabled as much as
it enabled because somebody came up with
a clever math hack that's not even that
accurate it's just accurate enough so as
is said here the wins we saw on compute
the revolution in 3D games that we saw
after that code came out and people
started using the engine that wasn't
because gpus or CPUs got way better it's
because our understanding of how to use
them to do these specific things got
better and we saw massive wins not
because the CPU got way faster but
because we found smarter ways to use it
and I think this is going to be true now
more than ever in the same way we're
reaching the cap of how much you can do
with a C CPU we're reaching the cap of
how much you can do with an llm
companies like open AI show that
focusing on more compute may still lead
to massive gains as compute power
despite the warning of Mo's law
continues to increase several orders of
magnitude over the next decades don't
necessarily agree currently the hype is
definitely outperforming Mo's law see
the image below as a result AI is at
risk of creating a deep environmental
footprint and research is increasingly
restricted to large corporations that
can afford to pay for the compute it's a
bitter lesson of the last year yeah this
is a fun one Mo's law versus AI
popularity but again Moors law is
plateauing and AI is now way more
popular than what Mor's law enables so
we're just spending billions on gpus
found a surprisingly good chart from
Gartner believe it or
not the hype cycle for artificial
intelligence hype Cycles are very
common this particular chart the startup
hype cycle an idea happens we have a
spike of excitement first Valley of
Death happens where you realize this is
hard you go hard you go really
hard you get inflated expectations
irrational exuberance and then pain you
end up in this thing called the trough
of disillusionment where you're unsure
of everything then the slow slope of
reality as you figure out what you're
actually capable of and what your
product company Vision whatever it is
actually could resolve to and then you
hit the real company and real value so
back to the Gartner chart it's funny
they have all these examples in here
first principles AI multi M agent
systems neuros symbolic AI more and more
things happening and we start getting
into generative Ai and then we hit a
massive Point realized we needed more
optimization things like synthetic data
better model optimization AI that is on
the edge so to speak so it runs on our
phones instead of on the servers
knowledge graphs but you notice we're
going down because these things aren't
fun these things suck and they're
necessary for us to keep evolving then
we started seeing AI makers and teaching
kits to try and get people to actually
learn autonomous vehicles with
which were very painful and still are
cars that drive themselves are far from
functioning but now we're seeing more
and more things that will hopefully
allow us to really benefit from AI but
we need to make sure our expectations
are realistically set not around the
exponential growth every year rather
around how we apply the functionality of
these things to actually benefit Our
Lives day in and day out I am honestly
just annoyed that people pretend the
models are going to get two times better
every couple years because we went
through that that's clearly over we're
just not seeing levels up like that
anymore what I'm expecting us to see
instead is massive winds in things that
we're not currently using models 4 like
we're starting to see video generation
catch on and it's taking us a lot of
time to get there but I could see us
growing there really quickly similar to
how chat GPT got way better really
quickly but it will also hit a plateau
and I think we're going to see more and
more of those plateaus hit and our
solution isn't going to be magically
make it better it's going to be entirely
different models and hybrids where we
take advantage of handwritten and
crafted code maybe human massaging of
things and AIS and intermingling and
mixing those the same way CPUs and gpus
take turns working on things depending
on what each is best at handwritten code
and AI code doing similar stuff has a
ton of potential and I think that's
going to be the future of AI because
this this is not the future of AI this
is a flatline this is a plateau This Is
Us reaching the end not the beginning
and if mistol is saying that their
model's large enough I'm inclined to
agree especially when you look at the
numbers here and you see how close all
of these models are getting to being
basically even the winds are no longer
the models being way better than the
others the winds are going to be
efficiency performance speed of
responses and then the next set of wins
is going to be how we use these things
in new and unique ways this is actually
a very interesting link there's a
project called The Arc prize that was
just linked from chat AGI progress has
stalled new ideas are needed
it's a million dooll public competition
to beat an open- Source a solution to
the ark AGI
Benchmark most AI benchmarks measure
skill but skill is not intelligence
general intelligence is the ability to
efficiently acquire new skills charlot's
unbeaten 2019 abstraction and reasoning
Corpus for artificial general
intelligence is the only formal
Benchmark of AGI it's easy for humans
but it's hard for
AI oh this is fun this is this is going
to be like captions basically so we have
these patterns an input and an output
it's pretty clear what we do here
configure your output grid there and
then we have to put the dark Blues here
here here and
here submit fun so the point here is
these are the types of puzzles that we
can Intuit it we look at the pattern and
we can learn quickly what the pattern is
with these things it looks like the
light blue is ignored
red has the outward pattern then dark
blue has the pattern with the like
t-shape but AI is historically really
bad at solving these types of things so
here's the arc AGI progress but if we
look at other AI benchmarks that people
use a lot of the ones we were looking at
earlier like H swag imag net all of
these it seems like things are improving
at an insane rate when you look at
general intelligence through a benchmark
like this AI sucks at it progress
towards artificial general intelligence
is stalled llms are trained on
unimaginably vast amounts of data yet
they remain unable to adapt to simple
problems they haven't been trained on or
make novel inventions no matter how
basic strong Market incentives have
pushed Frontier AI research to go closed
Source research attention and resources
are being put towards a dead end you can
change that I like that they have a they
call out that the consensus definition
for AGI is wrong AGI is a system that
can automate the majority of
economically valuable work but in
reality AGI is a system that can
efficiently acquire new skills and solve
open-ended problems yes that's what the
General in AGI stands for I actually
fully agree with this call out defition
are important because we turn them into
benchmarks to measure progress towards
AI I fully agree I love that Nat fredman
is one of the advisers the old CEO of
GitHub we also have Mike knop who's an
absolute Legend who's been involved in
all things software Dev and AI for a
very long time yeah I love this and I
think this is the only way we're going
to really see improvements and wins with
AI llms are hitting their limitations
and as we saw here they're not really
winning on General benchmarks like this
and sure we have these fancy benchmarks
that everybody loves but even these
we're starting to see a flatline and a
plateau on them we might be at the end
of the llm Revolution and if we want to
see AI continue to grow and advance in
its capabilities we might have to leave
behind llms the same way we're starting
to leave behind CPUs the future isn't an
llm but faster if we want the future to
be AI it has to be a different type of
AI let me know what you guys think and
tell me all the ways I'm wrong until
next time peace nerds
Weitere ähnliche Videos ansehen
How Far Can We Scale AI? Gen 3, Claude 3.5 Sonnet and AI Hype
The AI Hype is OVER! Have LLMs Peaked?
NO: GPT, Claude e gli altri NON SONO COSCIENTI. Propongo una soluzione.
So Google's Research Just Exposed OpenAI's Secrets (OpenAI o1-Exposed)
SHOCKING Robots EVOLVE in the SIMULATION plus OpenAI Leadership Just... LEAVES?
ChatGPT Can Now Talk Like a Human [Latest Updates]
5.0 / 5 (0 votes)