Amazon CEO's LEAKED Conversation Reveals Stunning Truth About The Future Of Software Engineering
Summary
TLDRThe script discusses the impact of AI on software development, referencing a leaked recording from Amazon Web Services' CEO, Matt Garman. It suggests that AI advancements could reduce the need for traditional coding by developers. The video aims to provide grounded insights into the potential shifts in the industry, highlighting the rapid progress in AI capabilities and the necessity for developers to adapt by upskilling and focusing on innovation and user experience. It also addresses the fear of job loss due to AI, emphasizing the optimistic view of new creative opportunities and the role changes rather than extinction.
Takeaways
- ๐ง The script discusses the rapid evolution of AI in software development and its potential impact on the job market, emphasizing that AI is not about to replace all software engineers but will change the nature of their work.
- ๐ข A leaked recording from Amazon Web Services CEO Matt Garman suggests that AI could take over many coding tasks, implying that developers might need to develop other skills as AI advances.
- ๐ฎ The prediction is made that within 24 months, most developers might not be coding as AI takes over, highlighting a significant shift in the industry within the next two years.
- ๐ AI's ability to perform coding tasks in natural language with high efficiency is a game-changer, potentially reducing the need for traditional coding by software engineers.
- ๐ค The role of software developers is expected to evolve, with a focus on innovation and building user-centric solutions rather than just writing code.
- ๐ The script references the rapid improvement in AI's coding capabilities, with benchmarks showing significant progress in a short time frame.
- ๐ The importance of understanding the fundamentals of coding and the underlying technology is underscored, as AI tools will still require knowledgeable users to operate effectively.
- ๐ก The video script suggests that the fear of AI replacing jobs is somewhat misplaced, as the technology is more likely to augment the role of software engineers rather than eliminate it.
- ๐ OpenAI's release of a human-validated subset of the SWE Bench indicates a commitment to improving the evaluation of AI models' ability to solve real-world software issues.
- ๐ The script points to the potential for AI to democratize programming, allowing domain experts to utilize technology without needing to be skilled programmers.
- ๐ The takeaway for aspiring software developers is to focus on developing a broad set of skills, including understanding AI and its applications in software engineering.
Q & A
What was the main point discussed in the leaked Amazon Cloud Chief's conversation?
-The main point was that AI could potentially take over many coding tasks, which might lead to a shift in the role of software developers, rather than completely replacing them.
What is the Amazon Web Services CEO Matt Garman's prediction regarding AI and coding?
-Matt Garman predicts that within 24 months, AI could take over many coding tasks, and software engineers might need to develop other skills as their traditional coding role changes.
What is the current capability of AI in coding tasks according to the transcript?
-AI is already capable of performing many coding tasks with remarkable efficiency in natural language, and this capability is expected to increase in the near future.
What does the term 'SWE Bench' refer to in the context of the script?
-SWE Bench is a benchmark for evaluating large language models' abilities to solve real-world software issues sourced from GitHub.
How has the performance of AI models on the SWE Bench evolved recently?
-The performance of AI models on the SWE Bench has significantly improved, with some models like Cosign Genie scoring as high as 43.8%, indicating rapid progress in AI's software engineering capabilities.
What does the transcript suggest about the future role of software engineers?
-The transcript suggests that the role of software engineers will change, with a focus on innovation, understanding customer needs, and managing AI systems rather than solely writing code.
What is the significance of the statement 'the hottest new programming language is English'?
-This statement highlights the idea that as AI systems become more adept at understanding natural language prompts, English could become the primary means of interacting with these systems for programming tasks.
How does the transcript address the fear of AI replacing jobs in the software industry?
-The transcript acknowledges the fear but emphasizes that AI is more likely to change the role of software engineers by automating routine coding tasks, allowing them to focus on more creative and user-centric work.
What is the potential impact of AI advancements on the hiring process in tech companies?
-The potential impact includes a shift in the skills required for software engineering roles, with a possible focus on managing AI systems and understanding customer needs rather than traditional coding skills.
What does the transcript suggest about the rate of improvement in AI's coding capabilities?
-The transcript suggests that the rate of improvement is accelerating, with significant monthly gains in performance on benchmarks like the SWE Bench, indicating a rapid advancement in AI's ability to perform coding tasks.
How does the transcript discuss the balance between AI hype and realistic expectations?
-The transcript attempts to provide a balanced view by acknowledging the rapid advancements in AI while also referencing skeptics who argue that some claims about AI capabilities may be overhyped or premature.
Outlines
๐ค AI's Impact on Software Engineering Jobs
The script discusses the evolving landscape of software development due to AI advancements. It references a leaked recording from Amazon Web Services' CEO, Matt Garman, who suggests that AI could soon take over many coding tasks, prompting a shift in the role of software engineers. The video aims to debunk hype and provide grounded analysis based on industry developments. Garman's comments are interpreted as an advisory nudge rather than a warning of job extinction, emphasizing the need for developers to adapt by developing other skills as AI continues to advance.
๐ฃ๏ธ The Emergence of English as the New Coding Language
This paragraph highlights the idea that English could become the primary means of interacting with AI systems, as AI's ability to understand and execute natural language prompts is improving. The role of software developers is expected to change, with a focus on understanding customer needs and innovating rather than writing code. The video script points out that AI tools could automate code generation, potentially reducing the demand for traditional coding skills but also creating opportunities for developers to upskill and work more closely with AI to enhance productivity.
๐ฎ Predictions and Speculations on AI in Coding
The script delves into predictions about the future of programming with AI, citing opinions from industry leaders like the CEO of Stability AI and Nvidia's CEO, who foresee a significant reduction in the need for human programmers. It emphasizes the importance of domain expertise and the potential for AI to democratize programming by enabling non-experts to solve complex problems. The video also addresses the rapid pace of AI development and the need for continuous learning and adaptation in the field of software engineering.
๐ Analysis of AI's Progress in Software Engineering Tasks
The paragraph presents an analysis of AI's capabilities in software engineering, particularly focusing on the performance of AI models on the Software Engineering Benchmark (SWE Bench). It discusses the rapid improvement in AI's ability to solve real-world software issues, as evidenced by the increasing scores on the SWE Bench. The video script also mentions OpenAI's release of a human-validated subset of the SWE Bench to more accurately evaluate AI models, suggesting that previous benchmarks may have underestimated AI's capabilities.
๐ Projecting AI's Future in Software Engineering
This section of the script extrapolates the current rate of improvement in AI to predict future capabilities in software engineering. It suggests that AI could be capable of solving a significant majority of software engineering tasks within the next 10 to 12 months, based on the current acceleration phase of technological progress. The video acknowledges the potential for this growth to slow as AI approaches theoretical limits but maintains that the next 24 to 48 months will see a substantial shift in the role of software engineers due to AI.
๐ Alpha Code 2's Achievements in Competitive Programming
The final paragraph discusses the achievements of Alpha Code 2, an AI system that has demonstrated high performance in competitive programming challenges. The system's ability to rank between expert and candidate master levels on Codeforces, a well-known competitive programming platform, indicates its advanced problem-solving capabilities. The video script uses this as an example to illustrate the potential future impact of AI on the field of software engineering, suggesting that AI could soon perform tasks typically expected of mid-level to senior software engineers.
Mindmap
Keywords
๐กAI
๐กSoftware Development
๐กCoding
๐กAmazon Web Services (AWS)
๐กNatural Language Processing (NLP)
๐กDevelopers
๐กInnovation
๐กAI Models
๐กUpskilling
๐กCompetitive Programming
Highlights
Amazon Web Services CEO Matt Garman suggests most developers may stop coding as AI takes over coding tasks.
AI's ability to perform coding tasks in natural language is speculated to replace many traditional coding jobs.
Garman's comments were advisory rather than a warning, indicating a shift in software engineering dynamics.
The role of software developers will change, with a focus on innovation and user experience rather than coding.
AI advancements are expected to automate code generation, reducing the need for manual coding by developers.
The prediction that most developers won't be coding in 24 months reflects the rapid pace of AI in software development.
Coding as a language to communicate with computers may become less central as AI takes on more coding tasks.
The potential for AI to generate entire programs from a single prompt signifies a major shift in software creation.
The hottest new programming language being English reflects the use of natural language prompts for AI coding.
Garman's optimistic tone suggests more creative opportunities for developers in the AI-enhanced landscape.
AWS is helping employees upskill to increase productivity with AI, showing a proactive approach to industry change.
The demand for software engineers may increase in the short term due to the need to understand and manage AI systems.
The role of a developer in 2025 may be significantly different, with a focus on managing AI rather than traditional coding.
The rapid improvement in AI coding capabilities, as seen in benchmarks, suggests an approaching inflection point in the industry.
OpenAI's release of a human-validated subset of the SWE Bench aims to more accurately evaluate AI's software engineering abilities.
The underestimation of AI capabilities by current benchmarks indicates that models may be more advanced than realized.
The improvements in AI coding agents' performance on the SWE Bench show an acceleration in the capabilities of AI in software engineering.
Estimations based on current improvement rates suggest AI could reach 90% of SWE tasks within the next 12 months.
The potential for AI to perform at a level comparable to expert competitive programmers indicates advanced problem-solving capabilities.
The discussion on the future of software engineering roles and the impact of AI on hiring processes and company structures.
Transcripts
so software development Ai and the job
market is rapidly evolving and in a
recent leaked recording an Amazon Cloud
Chief tells employees that most
developers could stop coding as soon as
AI takes over now I know what most of
you guys are initially thinking you're
probably thinking that this is an AI
hype video where I'm saying that oh no
all AI software Engineers are going to
replace all traditional software
engineer that's not what I'm saying I'm
going to break down this article and
show you the actual real grounded truth
that's based on ual industrywide
developments in the AI community that
most people aren't paying attention to
so this is an article from Business
Insider I find it quite insightful
because this is something that actually
happened in a leaked conversation with
the rate of current AI developments I'm
not sure that this person is entirely
wrong although I think there are a few
nuances that maybe this article doesn't
pay attention to so a summary of this
article basically says that the Amazon
web services CEO Matt garm shared
thoughts on AI during an internal
fireside chat in June and Business
Insider obtained a recording of the
meeting garmin's comments were a kind of
advisory nudge rather than a dire
warning to software Engineers hence the
part where I'm stating that this isn't a
kind of oh no all software Engineers are
gone but like I said before AI is most
certainly going to be changing the
dynamic so let's take a look at this and
see exactly what was said and what this
truly does mean and any other further
things for the industry so he says here
the software Engineers may need to
develop other skills as soon as
artificial intelligence takes over many
coding tasks if you aren't familiar with
the current concept of how good AI is
many people have been speculating that
AI is going to replace many coding tasks
due to its ability to perform many
coding tasks in natural language with a
remarkable level of efficiency now I do
think that sometime in the future this
is going to happen but there is a bit
more detail that you do need to pay
attention and it says that's according
to Amazon web services CEO Matt Garman
who shared his thoughts on the topic
during an internal fireside chat held in
June according to a recording of the
meeting obtained by Business Insider now
here's where he gives his prediction for
the dates in terms of where he thinks
this event will happen so he says that
if you go 24 months from now which is
literally just 2 years he says or some
amount of time I can't exactly predict
where it is but it is possible that most
developers are not coding okay and I
think what he says here is rather
accurate considering how people
interpret that comment okay so he says
here is that coding is just kind of like
that language where we talk to computers
it's not necessarily the skill in and of
itself the executive said he said that
skill in and of itself is like how do I
innovate how do I go build something
that's interesting for my end users to
use and I think this right here actually
captures what most people miss about
this when people say that okay software
engineering is is going to potentially
change considering the rate of tools and
advancements in the AI space and how
good these systems are getting at coding
related tasks and when we look into the
future we can kind of see that okay this
is clearly going to change the industry
now of course he does say here that 24
months from now things are going to look
different and I think 24 months from now
is not a bad estimate because 24 months
from now is 2 years currently it's 2024
that would be 2026 and in 24 number from
now in 2 years arguably there would have
at least been potentially two more
scale-ups of AI models now maybe there
might not be two more scale-ups of
course there are all of these things
that we cannot truly predict but I do
think 24 months from now the space might
be in a completely different position
now with that being said if we are in a
position where 24 months from now these
systems are absolutely amazing where you
can simply build products through
natural language prompts then it is
possible that most developers aren't
going to be coded during a time where
llms are doing the majority of the heavy
lifting so I think that that is an
accurate statement for 24 months from
now what he doesn't say and what people
might take away from this is that
developers are going to be useless and
all of their jobs are going to be gone
that's not what he's saying what he's
saying is that most of them aren't going
to be coding remember he said this is
more of like an advisory nudge rather
than a DI up warning now the thing here
as well is that coding is essentially
just a language that we actually talk to
computers and how we get them to do
exactly what we want and basically if we
think about like far into the future of
course it might not be 24 months from
now it might be 4 years from now but the
end goal is always how do we innovate
and of course how do we actually build
something that's interesting for my end
users to use I think this is mainly the
end goal for anyone who's using software
the end goal is always how can I make
this you know product better for my
users and how can I innovate within this
to make products that are actually good
for my users so I think that that is a
really important prediction now one of
the things that was recently said on
Twitter you know earlier last year was
the fact that the hottest new
programming language is English now this
you know saying that the hottest new
programming language is English is
basically referring to the fact that
English is what people use to talk to
llms and if you've been talking to llms
you'll know that when you talk to llms
they can manage to get a lot of your
understanding through natural language
you know sometimes you do have to do a
bit of extra prompting but as long as
you understand English you're going to
be working in a very easy environment
with these program now the article
continues to State some more things like
this role will change and he says the
role of the software developer will
change garment said it just means that
each of us has to get more in tune with
what our customers need and what the
actual end thing is that we're going to
try and go build okay because that's
going to be more and more of what work
is opposed to as of sitting down and
actually writing code and this is
something that I do agree with once the
AI is able to completely automate code
let's say you know 10 years from now ai
is just able to generate code for like
you know an entire program with one
single prompt which I think we're
starting to see you know early Sparks of
that with Claude 3.5 I think of course
the main thing you know the kind of
place that you want to be in is one
where you can actually think about what
the end user experience is going to be
like and what customers are actually
going to want of course it's not going
to be in you know doing the heavy
through code if llms are going to be
doing that the work is going to shift to
be you know actually understanding what
customers actually need and what the end
thing is so I think what he's stating
here is that you know this role is
probably not going to go away but you
know when you actually think about it
the role is going to change and it's
going to be really interesting to see
how the role manages to change when a
large portion of your work does get
automated I think it's going to be
interesting to see how individuals
manage to adapt to that changing work
environment and use any other skills in
order to adapt to the workplace now I do
think that this is going to be something
that is quite true of course he says
here that this is no dire warning of
course the talk of AI changing and even
eliminating jobs has intensified lately
as companies layoff employees or stop
hiring to shift resources towards AI
development new AI tools should
automatically generate code can help
companies do more with the same number
of Engineers or fewer of these pricey
employees if you aren't familiar with
the price of an you know software
engineer these guys are paid big big
bucks especially at Fan companies and
demand uh you know higher salary than
most traditional roles you can see here
it says in garment's case he was sharing
advice rather than issuing a dire
warning that software developers will go
extinct because of AI his tone is
optimistic suggesting more creative
opportunities for developers and he says
that AWS Amazon web services was helping
employees continue to upskill and learn
about new technologies to increase their
productivity with the help of a I now I
think this is a stellar statement
because there is a lot of fear right now
the fact that AI can do a lot of things
and it's advancing so rapidly the fact
that this could eliminate jobs is
certainly a fear amongst many which is
why of course I do have my community but
this is something that like I said
before is a role that I think it's going
to be enhanced by AI because the thing
is that right now what we're seeing is
we're seeing an influx of people that
are getting into code because of these
you know llms and these systems what we
have now is a place where you know you
can ask an llm to code something for you
completely basic but if you don't
understand how that code Works how you
can change that code what to kind of
prompt the llm you're still going to be
pretty stuck in a rudimentary manner
when you're trying to build something
and I think that is also going to
actually shortterm increase a demand for
software Engineers because there are
many people I know right now including
myself who are building certain things
you know experimenting with code that
truly haven't really done that before so
it's going to be kind of interesting to
see how that Dynamic manages to shift
and how companies manage to integrate
software Engineers as rather more than
you know coders now I guess you could
say orchestrators of you know pieces of
software as their main focus so you can
see right here he says being a developer
in 2025 may be different than what it
was as a developer in 2020 and I think
this is going to be you know rather true
as your role's main focus is probably
going to shift so that's a huge hint
towards any aspiring software developer
or someone who is a software developer
the kinds of things that you going to be
focusing now essentially he says here
that this is no more undifferentiated
heavy lifting see an Amazon web services
spokesperson Aisha Johnson told Business
Insider that garman's comments conveyed
opportunities for developers to
accomplish more than they do today with
new AI tools he added that there was no
indication he expected a decline in the
role of develop like I said before you
know these tools ideally we do want them
to do the heavy lifting which is going
to free up more time for tasks that do
matter such as actually thinking about
what the end user wants which means that
overall these experiences are going to
get better now one of the things I do
also want to talk about was the fact
that whilst the statement does come from
the Amazon web services Cloud Chief you
know him stating in a private chat it
does seem quite bad like oh this company
had this private chat and they were
behind closed doors saying that you know
AI could take over with coding he's not
the only person that has said this okay
you know stability AI that company the
CEO Imad mustac also predicted in 20203
that there will be no human programmers
in 5 years he based this off a
prediction you know on a few factors
including the GitHub data so 41% of all
code on GitHub is currently AI generated
and the fact that mustac believes that
chat GPT will be available on all mobile
phones without an now this is an AI
overview so I haven't fact checked all
this stuff but the point here is that
he's not the only person that has made
this prediction other people including
nvidia's CEO has also iterated and
spoken about this as well I want to say
something and it's it's going to sound
completely opposite of what people feel
over the course of the last 10 years 15
years um almost everybody who sits on a
stage like this would tell you it is
vital that your children learn computer
science um everybody should learn how to
program and in fact it's almost exactly
the opposite it is our job to create
computer technology such that nobody has
to program and that the programming
language is human everybody in the world
is now a programmer this is the miracle
of artificial intelligence the countries
the people that understand how to solve
a domain problem in digital biology or
in education of young people or in
manufacturing or in farming those people
who understand domain expertise now can
utilize technology that is readily
available to you you now have a computer
that will do what you tell it to do it
is vital that we upskill everyone and
the upskilling process I I believe will
be delightful surprising so yeah that's
Nvidia CEO talking about the future and
I think what he's stating here a lot of
people are thinking okay do I now switch
my career I think what he's more so
talking about is like you know people
who are just just starting and those who
are extremely young getting into the
space that by the time their career
matures the area is going to be a lot
different now I still think you're going
to need to you know know the
fundamentals behind the scenes and that
kind of stuff but some people disagree
now in this video what I wanted to do is
I wanted to keep this video as balanced
as possible because I know that there is
a large amount of people out there who
do believe that this is something that
is completely overhyped and I did see a
recent video that actually spoke about
this in a lot of depth there were two
videos that I did watch and I'm going to
mention those now because I always try
to you know understand where my biases
May lie if I'm someone who has a channel
in the AI space talking about the
technology there are incentives for me
to exaggerate AI claims however I'm not
that kind of person I understand that
there are nuances to things and that
sometimes the technology might not be as
great as it is Promised for example we
did have the Devon Saga where you know
the internet of bugs four months ago you
know a few weeks after Devon was
released he released this video a
25-minute investigation into the first
AI software engineer now essentially he
stated in this video that it was you
know an upwork lie and that you know
essentially Devon wasn't as good as they
claimed now that's completely
understandable a lot of the times with
technology demos things aren't as good
as the creators claim because they're
trying to drum up hype for their product
and it worked and the thing is is that
while yes potentially Devin was a system
that may have been a bit overhyped I
think the underlying message is still
quite true the tech in the space is
actually not overhyped at all and the
most surprising thing about this is that
the four months since the Devon system
was released you can see that this video
was literally four months ago there have
actually been numerous developments in
the AI software engineering space that
most people haven't paid attention Devon
actually managed to grab the you know
Collective consciousness of the software
engineering space on social media but
the other more incremental updates the
ones that actually matter and are slowly
moving upwards people haven't been
paying attention that's why I put here
that things move pretty quickly
especially in the AI space and I'm going
to show you guys what I'm talking about
so open AI decided to release the
software engineering bench they
introduced this on the 13th of August
2024 which is just around 10 days ago
now depending on when I release this
video so they said we're releasing a
human validated subset of the swe bench
that more reliably evaluates AI model's
ability to solve real world software
issues so they basically said that look
there are issues on this bench okay and
you guys can take a look at this here it
says that look all right one of the most
popular evaluation Suites for software
engineering is the swe bench a benchmark
for evaluating large language models
abilities to solve real world software
issues sought from GitHub The Benchmark
involves giving agents a code repository
and issue description and challenging
them now listen to this to generate a
patch that resolved the problem they
described by the issue now coding agents
have made impressive progress on the swe
bench with top scoring agents scoring
20% on the swe bench and 43% on the swe
bench light according to that
leaderboard as of August the 5th but
here's the kicker okay they said that
their testing identified some swe bench
tasks which may be hard or impossible
okay to solve so basically what they're
saying here is that look we made a
benchmark and when we looked at the one
that everyone's currently benchmarking
their systems on that Benchmark had some
issues that were way too hard or
completely impossible to solve leading
the swe bench to systematically
underestimate the model's autonomous
software engineering capabilities I'm
going to say that one more time open I
came out and said that that our testing
indicates that some swe bench talks are
impossible to solve leading to swe bench
systematically underestimating the the
model's autonomous software engineering
capability which basically means that
look we made benchmarks and we realized
that your ones were pretty impossible to
solve and you guys aren't truly
realizing how capable these models truly
are which is an issue because if you've
been paying attention to the space what
you'll know is that there have been
major major updates for example with
fine-tuning you can see that the actual
Improvement since Devon since 4 months
ago this has completely doubled guys and
the software engineering you know bench
you can see right here that since the
Devon area which was around 133% you can
see that that was around here during
that time in 4 months performance has
doubled okay we've had numerous
competitors come out of the works we've
had Amazon's Q developer agent get
38.8% we've got the factory code Droid
that are you know aiming to build you
know like an army of autonomous software
agents and then we of course recently
had cosign Genie which is now the
state-of-the-art model at 43. 8% on that
same Benchmark that Devon was on that
17% Benchmark so for those you know who
are saying that you know this one
doesn't work and this you know is awful
yada y y I would love to see what people
have to say about the actual
improvements because many people are now
dismissing these claims and many people
are now stating that look this thing
isn't good at all this isn't actually
any kind of reliable Improvement but if
we've just been paying attention to the
rate of improvement here we can see that
this is absolutely incredible now I've
made this table in Google and you can
see right here that the improvements
here are rather Stark we can see that
literally at the start of 2024 we were
at 7% you can literally just see right
here 7% and now we are 8 months through
2024 and we at
43.8% and remember Devon was only four
months ago you can see and in the four
months you know we've had you know 38 38
37 36 26 you know things have been
moving rather rather quickly so it's
important to know that now I did some
you know not testing but you know I
spoke to Claude I wanted to ask them how
quickly are we going to be moving
towards 90% because the reason I said
90% is because to go from like 40% to
90% is rather easy and that is going to
be an acceleratory period but to get
from 90% to like 99% is a lot harder
like making gains when you're already
you know 90% of the way there it becomes
you know exponentially harder to make
those additional gains so it says here
that there is a significant jump from
the older models below 10% to the newer
ones about 20% the Improvement seems to
be accelerating with more recent models
showing larger gains this is for many
factors including the fact that you know
all these com companies are now coming
out of stealth showing you know everyone
what they've been building and it says
here that given these observations we
can make a rough estimate the
improvement from the best 2023 model
which is retrievable augmented
generation and Claw 3 Opus to the best
2024 model is about a
36.8% % increase in roughly 8 months
this suggests an average Improvement
rate of about 4.6 percentage points per
month to get from 43.8% to 90% we need
an additional 46.2 percentage points at
the current rate of improvement it would
have take approximately just 10 months
to reach 90% however technological
progress often follows an S curve where
improvements accelerate for a while then
slow down as we approach theoretical
limits and we're likely in the
acceleration phase now considering this
and the rapid re progress I would
estimate that reaching 90% could be
possible with 6 to 12 months from the
latest data in the date placing the
prediction sometime between February and
August
2025 so with Claude 3.5 doing the
analysis here you can see that if we
actually look at the data and we you
know extrapolate out further and say
okay you know we know what's going to
happen Claude is basically predicting
that look within 10 months you know it's
going to be 90% of all swe tasks that
are quite possible and then you know
Amazon Cloud Chief is basically saying
that look you know in the 24 months to
come things are going to be rather
different it doesn't seem that crazy
when you actually break it down number
by number and this is some important
things that you do have to pay attention
to especially if you're in this space
now one of the things that most people
do actually forget and I do think that
this prediction isn't overestimated this
isn't like a hype video where I'm like
oh my God you know AGI and you know 10
months or two months this is more of
like you know trying to keep it factual
and grounded literally based on the you
know benchmarks that we've recently seen
but we've seen like 4.6 percentage
Points each month of course some months
are going to be larger some months are
going to be not as good but I think if
you know if we go at the same rate it's
going to be 10 months we can you know
say that okay even if it's not 10 months
even if we extrapolate and add another
year onto that there's going to be huge
amounts of significant developments now
like I said before there's always one
thing that most people forget and this
is where I think there's going to be you
know huge jumps made in AIS coding
capabilities you know in the areas of
these Frontier you know models doing
stuff that just most people didn't even
take into account okay and that's why I
said the recent videos I'm talking about
you know such as this one right here you
know it's saying that debunking the AI
software engineer yada yada yada this is
terrible always take into account the
actual benchmarks of other AI systems
because that is also important and I did
watch this video from n code. where he's
basically explaining that look hype is a
marketing tool and basically stating
that hype is completely out of control
and that this is you know something that
could potentially not get there just yet
but I do think that what this video
doesn't pay attention to is of course
some of the stuff and of course some of
the more recent developments and some of
the other papers where they actually
talk about other coding stuff which I'm
going to get into in a minute most
powerful marketing tool that's ever
existed let's talk Devon AI for a second
I made a video talking about it right
after the announcement I thought the
founders were super smart but also that
I'm not worried about Devon because
these people are clearly fighting an
uphill battle
I don't know a lot about it the
benchmarks say that this is more
effective at software engineering tasks
than the other llms I wonder if that's
because it just puts a few things
together like it has its own
capabilities to like do research like go
on Google brows stack Overflow and run
code and execute code and they just put
those pieces together more more
cohesively than GPT obviously in a short
amount of time they didn't create their
own complex llm I'm pretty sure they're
using one of the existing llms I didn't
closely examine the evidence or anything
like that it seemed kind of obvious to
me a few months later people are
realizing they were fooled by the hype
Devon is extremely overrated at least
for now if you don't believe me maybe
you'll believe this guy he basically
proved that at this point Devon AI is
more useless than a freshman CS student
who's one week into their first
programming course so what I wanted to
talk about you know after you know
viewing ni's video it was definitely an
insightful video and I do think that it
grounds you know a lot more you know
reality in terms of what's actually
happening because I think what we do
need to pay attention to is the reality
of things you know a lot of things have
blown out in that video he talks about
Tesla's been selling for self driving
for years you know companies make claims
all the time you've got FTX you've got
theranos this is something that usually
does happen with tech companies but most
people aren't paying attention to what
these Frontier companies are saying okay
for example um this is a paper that
didn't get a lot of you know information
slash like data you know people just
basically didn't speak about this paper
but essentially this is the alpha code 2
technical report okay and it talks about
how alpha code was the first AI system
to perform at the level of the median
competitor in competitive programming a
difficult reasoning tasks involving
Advanced maths logic and computer
science and this paper introduces Alpha
code 2 a new and enhanced system with
massively improved performance powered
by Gemini Alpha code 2 relies on the
combination of powerful LMS and a
bespoke searching and reranking
reckoners when evaluated on the same
platform as the original Alpha code we
found that Alpha code 2 solved 1.7 times
more problems and performed better than
85% of competition participants now I'm
just going to you know ground this in
you know what you can understand here
for including myself we can see here
that uh basically this was done on the
code forces okay and this was able to
get the 85 percentile on the code forces
so code forces is a wellknown platform
for competitive programming with a
ranking system that categorizes
participants based on their performance
in the contest the ranks include newbie
pupil specialist expert candidate master
master International Master grandmas
International Grandmaster legendary
Grandmaster and reaching 85th percentile
on code forces means performing better
than 85% of all competitors and
according to the data Alpha code 2 ranks
between the expert and candidate Master
levels which were quite Advanced now if
you want to ground that in terms of how
good it could potentially show us how
you know these systems are going to be
in terms of you know coding abilities
and what we can look at for the future
you can see here that for code forces
the expert level which is you know the
top 70% to 85% participants at this
level have a strong understanding of
algorithms and data structures they're
proficient in solving standard
competitive programming problems and are
beginning to solve more challenging
problems that require deeper insights or
more Advanced Techniques and the
comparison to software engineering roles
here basically says that typically
focusing on learning and applying
basically programming Concepts writing
clean code contributing to projects
under supervision the main problem
solving required for competitive
programming at the expert or candidate
level is generally beyond what is
expected of a junior software engineer
and this is usually what you get at a
midlevel to senior level software
engineer and engineers at this level are
expected to have strong problemsolving
skills understanding algorithms and data
structures and be capable of Designing
efficient and scalable solutions they
might not participate in competitive
programming but they should be
comfortable with similar challenges when
needed the point here guys is that what
we have is a scenario where you know we
should discount the technology as for
what we've seen recently as that every
single month this is going to you know
continually increase one of the things
that I think most people aren't paying
attention to is the fact that you know
even open AI have openly said that this
is going to be something themselves that
they are going to be tackling for
example take a look at this clip of
samman actually talking about what one
of the biggest areas of improvement is
likely going to be for him moving
forward more to gain there graph a
couple of decades in the future be like
H something changed yeah are there
application
or areas you think are most promising in
the next 12 months I'm sure I'm biased
just because of where what we do here
but Cod I think is a it's a really big
one and the only reason I include that
clip is because opening eye have
previously always surprised us with
their capabilities and you know they
tend to surprise us in areas that we we
aren't even pretty much focusing on so
the takeaways are simple guys what we
have here is a situation where you know
over the next 24 to 48 months we could
see a complete shift in terms of the
main role of software Engineers I still
think that we're going to struggle to
get from 90% all the way to you know 99%
so I think that there will be like this
scurve growth where you do have a
situation where right now you know if
you look at the benchmarks we are
currently during that acceleration
period where you know you've got these
systems you know at the start it's
really hard to make progress then all of
a sudden the jumps happen the jumps
happen then as you get to like 90 95%
that's where the taper off is so I do
think that if you're like a senior level
software engineer to someone that's
really cracked I do think that you know
your job is probably not going to change
for the most part because you're still
going to have to understand how all of
those systems work together and some of
the most difficult tasks but I do think
that you know as time goes on you know
AI is probably going to eat like the
bottom area where you're going to be
writing code and you know as time goes
on it's going to continue continue to
continue to continue to you know shift
the role of you know many software
Engineers I think one of the most
interesting things from all of this is
how companies are going to change in
terms of their hiring process is someone
just going to be you know managing AI
agents is it going to be a new you know
AI agent structure that allows you to
you know manage the entire code base
you've got AI systems that are going to
be working with millions and millions of
context lengths either way I think this
is something that you know is going to
really really change in the next two to
four years arguably one of the biggest
changes that's going to happen but
either way I don't think it's all doom
and gloom I just think that the role
will change and I think it's going to be
fascinating to see how that happens if
you did enjoy the video don't forget to
leave a like do forget to subscribe and
I'll see you guys in the next one
Browse More Related Video
Amazon's LEAKED Conversation Reveals Stunning Truth About The Future Of Software Engineering
AWS CEO - The End Of Programmers Is Near
GitHub's Devin Competitor, Sam Altman Talks GPT-5 and AGI, Amazon Q, Rabbit R1 Hacked (AI News)
Which Career Path To Choose? Future Scope? Salaries? Job Opportunities?๐ค Computer Science Students
The Future of AI ( Artificial Intelligence)
Andrew Ng on AI's Potential Effect on the Labor Force | WSJ
5.0 / 5 (0 votes)