Amazon CEO's LEAKED Conversation Reveals Stunning Truth About The Future Of Software Engineering

Summary

The script discusses the impact of AI on software development, referencing a leaked recording from Amazon Web Services' CEO, Matt Garman. It suggests that AI advancements could reduce the need for traditional coding by developers. The video aims to provide grounded insights into the potential shifts in the industry, highlighting the rapid progress in AI capabilities and the necessity for developers to adapt by upskilling and focusing on innovation and user experience. It also addresses the fear of job loss due to AI, emphasizing the optimistic view of new creative opportunities and the role changes rather than extinction.

Takeaways

• 🧠 The script discusses the rapid evolution of AI in software development and its potential impact on the job market, emphasizing that AI is not about to replace all software engineers but will change the nature of their work.
• 📢 A leaked recording from Amazon Web Services CEO Matt Garman suggests that AI could take over many coding tasks, implying that developers might need to develop other skills as AI advances.
• 🔮 The prediction is made that within 24 months, most developers might not be coding as AI takes over, highlighting a significant shift in the industry within the next two years.
• 🛠 AI's ability to perform coding tasks in natural language with high efficiency is a game-changer, potentially reducing the need for traditional coding by software engineers.
• 🤖 The role of software developers is expected to evolve, with a focus on innovation and building user-centric solutions rather than just writing code.
• 📈 The script references the rapid improvement in AI's coding capabilities, with benchmarks showing significant progress in a short time frame.
• 📚 The importance of understanding the fundamentals of coding and the underlying technology is underscored, as AI tools will still require knowledgeable users to operate effectively.
• 💡 The video script suggests that the fear of AI replacing jobs is somewhat misplaced, as the technology is more likely to augment the role of software engineers rather than eliminate it.
• 🌐 OpenAI's release of a human-validated subset of the SWE Bench indicates a commitment to improving the evaluation of AI models' ability to solve real-world software issues.
• 🚀 The script points to the potential for AI to democratize programming, allowing domain experts to utilize technology without needing to be skilled programmers.
• 🔑 The takeaway for aspiring software developers is to focus on developing a broad set of skills, including understanding AI and its applications in software engineering.

Q & A

What was the main point discussed in the leaked Amazon Cloud Chief's conversation?

The main point was that AI could potentially take over many coding tasks, which might lead to a shift in the role of software developers, rather than completely replacing them.

What is the Amazon Web Services CEO Matt Garman's prediction regarding AI and coding?

Matt Garman predicts that within 24 months, AI could take over many coding tasks, and software engineers might need to develop other skills as their traditional coding role changes.

What is the current capability of AI in coding tasks according to the transcript?

AI is already capable of performing many coding tasks with remarkable efficiency in natural language, and this capability is expected to increase in the near future.

What does the term 'SWE Bench' refer to in the context of the script?

SWE Bench is a benchmark for evaluating large language models' abilities to solve real-world software issues sourced from GitHub.

How has the performance of AI models on the SWE Bench evolved recently?

The performance of AI models on the SWE Bench has significantly improved, with some models like Cosign Genie scoring as high as 43.8%, indicating rapid progress in AI's software engineering capabilities.

What does the transcript suggest about the future role of software engineers?

The transcript suggests that the role of software engineers will change, with a focus on innovation, understanding customer needs, and managing AI systems rather than solely writing code.

What is the significance of the statement 'the hottest new programming language is English'?

This statement highlights the idea that as AI systems become more adept at understanding natural language prompts, English could become the primary means of interacting with these systems for programming tasks.

How does the transcript address the fear of AI replacing jobs in the software industry?

The transcript acknowledges the fear but emphasizes that AI is more likely to change the role of software engineers by automating routine coding tasks, allowing them to focus on more creative and user-centric work.

What is the potential impact of AI advancements on the hiring process in tech companies?

The potential impact includes a shift in the skills required for software engineering roles, with a possible focus on managing AI systems and understanding customer needs rather than traditional coding skills.

What does the transcript suggest about the rate of improvement in AI's coding capabilities?

The transcript suggests that the rate of improvement is accelerating, with significant monthly gains in performance on benchmarks like the SWE Bench, indicating a rapid advancement in AI's ability to perform coding tasks.

How does the transcript discuss the balance between AI hype and realistic expectations?

The transcript attempts to provide a balanced view by acknowledging the rapid advancements in AI while also referencing skeptics who argue that some claims about AI capabilities may be overhyped or premature.

Outlines

00:00

🤖 AI's Impact on Software Engineering Jobs

The script discusses the evolving landscape of software development due to AI advancements. It references a leaked recording from Amazon Web Services' CEO, Matt Garman, who suggests that AI could soon take over many coding tasks, prompting a shift in the role of software engineers. The video aims to debunk hype and provide grounded analysis based on industry developments. Garman's comments are interpreted as an advisory nudge rather than a warning of job extinction, emphasizing the need for developers to adapt by developing other skills as AI continues to advance.

05:02

🗣️ The Emergence of English as the New Coding Language

This paragraph highlights the idea that English could become the primary means of interacting with AI systems, as AI's ability to understand and execute natural language prompts is improving. The role of software developers is expected to change, with a focus on understanding customer needs and innovating rather than writing code. The video script points out that AI tools could automate code generation, potentially reducing the demand for traditional coding skills but also creating opportunities for developers to upskill and work more closely with AI to enhance productivity.

10:03

🔮 Predictions and Speculations on AI in Coding

The script delves into predictions about the future of programming with AI, citing opinions from industry leaders like the CEO of Stability AI and Nvidia's CEO, who foresee a significant reduction in the need for human programmers. It emphasizes the importance of domain expertise and the potential for AI to democratize programming by enabling non-experts to solve complex problems. The video also addresses the rapid pace of AI development and the need for continuous learning and adaptation in the field of software engineering.

15:04

📊 Analysis of AI's Progress in Software Engineering Tasks

The paragraph presents an analysis of AI's capabilities in software engineering, particularly focusing on the performance of AI models on the Software Engineering Benchmark (SWE Bench). It discusses the rapid improvement in AI's ability to solve real-world software issues, as evidenced by the increasing scores on the SWE Bench. The video script also mentions OpenAI's release of a human-validated subset of the SWE Bench to more accurately evaluate AI models, suggesting that previous benchmarks may have underestimated AI's capabilities.

20:05

📈 Projecting AI's Future in Software Engineering

This section of the script extrapolates the current rate of improvement in AI to predict future capabilities in software engineering. It suggests that AI could be capable of solving a significant majority of software engineering tasks within the next 10 to 12 months, based on the current acceleration phase of technological progress. The video acknowledges the potential for this growth to slow as AI approaches theoretical limits but maintains that the next 24 to 48 months will see a substantial shift in the role of software engineers due to AI.

25:06

🏆 Alpha Code 2's Achievements in Competitive Programming

The final paragraph discusses the achievements of Alpha Code 2, an AI system that has demonstrated high performance in competitive programming challenges. The system's ability to rank between expert and candidate master levels on Codeforces, a well-known competitive programming platform, indicates its advanced problem-solving capabilities. The video script uses this as an example to illustrate the potential future impact of AI on the field of software engineering, suggesting that AI could soon perform tasks typically expected of mid-level to senior software engineers.

Keywords

💡AI

AI, or Artificial Intelligence, refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions. In the context of the video, AI is central to the discussion about the future of software development. The video discusses how AI advancements might lead to changes in the tasks performed by software engineers, potentially reducing the need for manual coding.

💡Software Development

Software Development is the process of creating and maintaining applications and systems. It involves coding, testing, and debugging. The video script suggests that AI could revolutionize this field by taking over many coding tasks, which would shift the focus of software engineers towards other aspects of development, such as innovation and user experience.

💡Coding

Coding is the act of writing source code in a particular programming language. The video discusses the possibility that AI could replace or reduce the amount of coding done by humans, as AI systems become more proficient at generating code from natural language prompts.

💡Amazon Web Services (AWS)

Amazon Web Services is a subsidiary of Amazon that provides on-demand cloud computing platforms and APIs to individuals, companies, and governments. In the video, AWS CEO Matt Garman's comments on AI's impact on software development are highlighted, suggesting that AWS is actively considering and preparing for AI's role in the future of coding.

💡Natural Language Processing (NLP)

Natural Language Processing is a subfield of AI that focuses on the interaction between computers and humans through natural language. The video mentions that AI's ability to understand and generate code from natural language is a significant factor in its potential to change software development.

💡Developers

Developers, or software developers, are professionals who create software applications and systems. The video script discusses the potential shift in the role of developers due to AI, suggesting that they may need to develop other skills as AI takes over many coding tasks.

💡Innovation

Innovation refers to the process of translating an idea or invention into a good or service that creates value or for which customers will pay. The video suggests that as AI takes over routine coding tasks, developers will focus more on innovation, creating new and interesting applications for end users.

💡AI Models

AI models are algorithms, often machine learning models, that are trained to perform tasks such as image recognition, language translation, or in this case, coding. The video discusses the rapid improvement in AI models' abilities to perform software engineering tasks, indicating a potential future where these models can handle more complex coding challenges.

💡Upskilling

Upskilling is the process of learning new skills or improving existing ones to meet new job requirements. The video mentions that AWS is helping employees upskill to adapt to the changes AI will bring to the software development industry.

💡Competitive Programming

Competitive Programming is a activity where programmers try to solve problems in a fixed amount of time, often in a contest format. The video references Alpha Code 2's performance in competitive programming as an indicator of AI's growing capabilities in complex coding tasks, suggesting that AI is becoming capable of performing at a level comparable to advanced human programmers.

Highlights

Amazon Web Services CEO Matt Garman suggests most developers may stop coding as AI takes over coding tasks.

AI's ability to perform coding tasks in natural language is speculated to replace many traditional coding jobs.

Garman's comments were advisory rather than a warning, indicating a shift in software engineering dynamics.

The role of software developers will change, with a focus on innovation and user experience rather than coding.

AI advancements are expected to automate code generation, reducing the need for manual coding by developers.

The prediction that most developers won't be coding in 24 months reflects the rapid pace of AI in software development.

Coding as a language to communicate with computers may become less central as AI takes on more coding tasks.

The potential for AI to generate entire programs from a single prompt signifies a major shift in software creation.

The hottest new programming language being English reflects the use of natural language prompts for AI coding.

Garman's optimistic tone suggests more creative opportunities for developers in the AI-enhanced landscape.

AWS is helping employees upskill to increase productivity with AI, showing a proactive approach to industry change.

The demand for software engineers may increase in the short term due to the need to understand and manage AI systems.

The role of a developer in 2025 may be significantly different, with a focus on managing AI rather than traditional coding.

The rapid improvement in AI coding capabilities, as seen in benchmarks, suggests an approaching inflection point in the industry.

OpenAI's release of a human-validated subset of the SWE Bench aims to more accurately evaluate AI's software engineering abilities.

The underestimation of AI capabilities by current benchmarks indicates that models may be more advanced than realized.

The improvements in AI coding agents' performance on the SWE Bench show an acceleration in the capabilities of AI in software engineering.

Estimations based on current improvement rates suggest AI could reach 90% of SWE tasks within the next 12 months.

The potential for AI to perform at a level comparable to expert competitive programmers indicates advanced problem-solving capabilities.

The discussion on the future of software engineering roles and the impact of AI on hiring processes and company structures.

Transcripts

so software development Ai and the job

market is rapidly evolving and in a

recent leaked recording an Amazon Cloud

Chief tells employees that most

developers could stop coding as soon as

AI takes over now I know what most of

you guys are initially thinking you're

probably thinking that this is an AI

hype video where I'm saying that oh no

all AI software Engineers are going to

replace all traditional software

engineer that's not what I'm saying I'm

going to break down this article and

show you the actual real grounded truth

that's based on ual industrywide

developments in the AI community that

most people aren't paying attention to

so this is an article from Business

Insider I find it quite insightful

because this is something that actually

happened in a leaked conversation with

the rate of current AI developments I'm

not sure that this person is entirely

wrong although I think there are a few

nuances that maybe this article doesn't

pay attention to so a summary of this

article basically says that the Amazon

web services CEO Matt garm shared

thoughts on AI during an internal

fireside chat in June and Business

Insider obtained a recording of the

meeting garmin's comments were a kind of

advisory nudge rather than a dire

warning to software Engineers hence the

part where I'm stating that this isn't a

kind of oh no all software Engineers are

gone but like I said before AI is most

certainly going to be changing the

dynamic so let's take a look at this and

see exactly what was said and what this

truly does mean and any other further

things for the industry so he says here

the software Engineers may need to

develop other skills as soon as

artificial intelligence takes over many

coding tasks if you aren't familiar with

the current concept of how good AI is

many people have been speculating that

AI is going to replace many coding tasks

due to its ability to perform many

coding tasks in natural language with a

remarkable level of efficiency now I do

think that sometime in the future this

is going to happen but there is a bit

more detail that you do need to pay

attention and it says that's according

to Amazon web services CEO Matt Garman

who shared his thoughts on the topic

during an internal fireside chat held in

June according to a recording of the

meeting obtained by Business Insider now

here's where he gives his prediction for

the dates in terms of where he thinks

this event will happen so he says that

if you go 24 months from now which is

literally just 2 years he says or some

amount of time I can't exactly predict

where it is but it is possible that most

developers are not coding okay and I

think what he says here is rather

accurate considering how people

interpret that comment okay so he says

here is that coding is just kind of like

that language where we talk to computers

it's not necessarily the skill in and of

itself the executive said he said that

skill in and of itself is like how do I

innovate how do I go build something

that's interesting for my end users to

use and I think this right here actually

captures what most people miss about

this when people say that okay software

engineering is is going to potentially

change considering the rate of tools and

advancements in the AI space and how

good these systems are getting at coding

related tasks and when we look into the

future we can kind of see that okay this

is clearly going to change the industry

now of course he does say here that 24

months from now things are going to look

different and I think 24 months from now

is not a bad estimate because 24 months

from now is 2 years currently it's 2024

that would be 2026 and in 24 number from

now in 2 years arguably there would have

at least been potentially two more

scale-ups of AI models now maybe there

might not be two more scale-ups of

course there are all of these things

that we cannot truly predict but I do

think 24 months from now the space might

be in a completely different position

now with that being said if we are in a

position where 24 months from now these

systems are absolutely amazing where you

can simply build products through

natural language prompts then it is

possible that most developers aren't

going to be coded during a time where

llms are doing the majority of the heavy

lifting so I think that that is an

accurate statement for 24 months from

now what he doesn't say and what people

might take away from this is that

developers are going to be useless and

all of their jobs are going to be gone

that's not what he's saying what he's

saying is that most of them aren't going

to be coding remember he said this is

more of like an advisory nudge rather

than a DI up warning now the thing here

as well is that coding is essentially

just a language that we actually talk to

computers and how we get them to do

exactly what we want and basically if we

think about like far into the future of

course it might not be 24 months from

now it might be 4 years from now but the

end goal is always how do we innovate

and of course how do we actually build

something that's interesting for my end

users to use I think this is mainly the

end goal for anyone who's using software

the end goal is always how can I make

this you know product better for my

users and how can I innovate within this

to make products that are actually good

for my users so I think that that is a

really important prediction now one of

the things that was recently said on

Twitter you know earlier last year was

the fact that the hottest new

programming language is English now this

you know saying that the hottest new

programming language is English is

basically referring to the fact that

English is what people use to talk to

llms and if you've been talking to llms

you'll know that when you talk to llms

they can manage to get a lot of your

understanding through natural language

you know sometimes you do have to do a

bit of extra prompting but as long as

you understand English you're going to

be working in a very easy environment

with these program now the article

continues to State some more things like

this role will change and he says the

role of the software developer will

change garment said it just means that

each of us has to get more in tune with

what our customers need and what the

actual end thing is that we're going to

try and go build okay because that's

going to be more and more of what work

is opposed to as of sitting down and

actually writing code and this is

something that I do agree with once the

AI is able to completely automate code

let's say you know 10 years from now ai

is just able to generate code for like

you know an entire program with one

single prompt which I think we're

starting to see you know early Sparks of

that with Claude 3.5 I think of course

the main thing you know the kind of

place that you want to be in is one

where you can actually think about what

the end user experience is going to be

like and what customers are actually

going to want of course it's not going

to be in you know doing the heavy

through code if llms are going to be

doing that the work is going to shift to

be you know actually understanding what

customers actually need and what the end

thing is so I think what he's stating

here is that you know this role is

probably not going to go away but you

know when you actually think about it

the role is going to change and it's

going to be really interesting to see

how the role manages to change when a

large portion of your work does get

automated I think it's going to be

interesting to see how individuals

manage to adapt to that changing work

environment and use any other skills in

order to adapt to the workplace now I do

think that this is going to be something

that is quite true of course he says

here that this is no dire warning of

course the talk of AI changing and even

eliminating jobs has intensified lately

as companies layoff employees or stop

hiring to shift resources towards AI

development new AI tools should

automatically generate code can help

companies do more with the same number

of Engineers or fewer of these pricey

employees if you aren't familiar with

the price of an you know software

engineer these guys are paid big big

bucks especially at Fan companies and

demand uh you know higher salary than

most traditional roles you can see here

it says in garment's case he was sharing

advice rather than issuing a dire

warning that software developers will go

extinct because of AI his tone is

optimistic suggesting more creative

opportunities for developers and he says

that AWS Amazon web services was helping

employees continue to upskill and learn

about new technologies to increase their

productivity with the help of a I now I

think this is a stellar statement

because there is a lot of fear right now

the fact that AI can do a lot of things

and it's advancing so rapidly the fact

that this could eliminate jobs is

certainly a fear amongst many which is

why of course I do have my community but

this is something that like I said

before is a role that I think it's going

to be enhanced by AI because the thing

is that right now what we're seeing is

we're seeing an influx of people that

are getting into code because of these

you know llms and these systems what we

have now is a place where you know you

can ask an llm to code something for you

completely basic but if you don't

understand how that code Works how you

can change that code what to kind of

prompt the llm you're still going to be

pretty stuck in a rudimentary manner

when you're trying to build something

and I think that is also going to

actually shortterm increase a demand for

software Engineers because there are

many people I know right now including

myself who are building certain things

you know experimenting with code that

truly haven't really done that before so

it's going to be kind of interesting to

see how that Dynamic manages to shift

and how companies manage to integrate

software Engineers as rather more than

you know coders now I guess you could

say orchestrators of you know pieces of

software as their main focus so you can

see right here he says being a developer

in 2025 may be different than what it

was as a developer in 2020 and I think

this is going to be you know rather true

as your role's main focus is probably

going to shift so that's a huge hint

towards any aspiring software developer

or someone who is a software developer

the kinds of things that you going to be

focusing now essentially he says here

that this is no more undifferentiated

heavy lifting see an Amazon web services

spokesperson Aisha Johnson told Business

Insider that garman's comments conveyed

opportunities for developers to

accomplish more than they do today with

new AI tools he added that there was no

indication he expected a decline in the

role of develop like I said before you

know these tools ideally we do want them

to do the heavy lifting which is going

to free up more time for tasks that do

matter such as actually thinking about

what the end user wants which means that

overall these experiences are going to

get better now one of the things I do

also want to talk about was the fact

that whilst the statement does come from

the Amazon web services Cloud Chief you

know him stating in a private chat it

does seem quite bad like oh this company

had this private chat and they were

behind closed doors saying that you know

AI could take over with coding he's not

the only person that has said this okay

you know stability AI that company the

CEO Imad mustac also predicted in 20203

that there will be no human programmers

in 5 years he based this off a

prediction you know on a few factors

including the GitHub data so 41% of all

code on GitHub is currently AI generated

and the fact that mustac believes that

chat GPT will be available on all mobile

phones without an now this is an AI

overview so I haven't fact checked all

this stuff but the point here is that

he's not the only person that has made

this prediction other people including

nvidia's CEO has also iterated and

spoken about this as well I want to say

something and it's it's going to sound

completely opposite of what people feel

over the course of the last 10 years 15

years um almost everybody who sits on a

stage like this would tell you it is

vital that your children learn computer

science um everybody should learn how to

program and in fact it's almost exactly

the opposite it is our job to create

computer technology such that nobody has

to program and that the programming

language is human everybody in the world

is now a programmer this is the miracle

of artificial intelligence the countries

the people that understand how to solve

a domain problem in digital biology or

in education of young people or in

manufacturing or in farming those people

who understand domain expertise now can

utilize technology that is readily

available to you you now have a computer

that will do what you tell it to do it

is vital that we upskill everyone and

the upskilling process I I believe will

be delightful surprising so yeah that's

Nvidia CEO talking about the future and

I think what he's stating here a lot of

people are thinking okay do I now switch

my career I think what he's more so

talking about is like you know people

who are just just starting and those who

are extremely young getting into the

space that by the time their career

matures the area is going to be a lot

different now I still think you're going

to need to you know know the

fundamentals behind the scenes and that

kind of stuff but some people disagree

now in this video what I wanted to do is

I wanted to keep this video as balanced

as possible because I know that there is

a large amount of people out there who

do believe that this is something that

is completely overhyped and I did see a

recent video that actually spoke about

this in a lot of depth there were two

videos that I did watch and I'm going to

mention those now because I always try

to you know understand where my biases

May lie if I'm someone who has a channel

in the AI space talking about the

technology there are incentives for me

to exaggerate AI claims however I'm not

that kind of person I understand that

there are nuances to things and that

sometimes the technology might not be as

great as it is Promised for example we

did have the Devon Saga where you know

the internet of bugs four months ago you

know a few weeks after Devon was

released he released this video a

25-minute investigation into the first

AI software engineer now essentially he

stated in this video that it was you

know an upwork lie and that you know

essentially Devon wasn't as good as they

claimed now that's completely

understandable a lot of the times with

technology demos things aren't as good

as the creators claim because they're

trying to drum up hype for their product

and it worked and the thing is is that

while yes potentially Devin was a system

that may have been a bit overhyped I

think the underlying message is still

quite true the tech in the space is

actually not overhyped at all and the

most surprising thing about this is that

the four months since the Devon system

was released you can see that this video

was literally four months ago there have

actually been numerous developments in

the AI software engineering space that

most people haven't paid attention Devon

actually managed to grab the you know

Collective consciousness of the software

engineering space on social media but

the other more incremental updates the

ones that actually matter and are slowly

moving upwards people haven't been

paying attention that's why I put here

that things move pretty quickly

especially in the AI space and I'm going

to show you guys what I'm talking about

so open AI decided to release the

software engineering bench they

introduced this on the 13th of August

2024 which is just around 10 days ago

now depending on when I release this

video so they said we're releasing a

human validated subset of the swe bench

that more reliably evaluates AI model's

ability to solve real world software

issues so they basically said that look

there are issues on this bench okay and

you guys can take a look at this here it

says that look all right one of the most

popular evaluation Suites for software

engineering is the swe bench a benchmark

for evaluating large language models

abilities to solve real world software

issues sought from GitHub The Benchmark

involves giving agents a code repository

and issue description and challenging

them now listen to this to generate a

patch that resolved the problem they

described by the issue now coding agents

have made impressive progress on the swe

bench with top scoring agents scoring

20% on the swe bench and 43% on the swe

bench light according to that

leaderboard as of August the 5th but

here's the kicker okay they said that

their testing identified some swe bench

tasks which may be hard or impossible

okay to solve so basically what they're

saying here is that look we made a

benchmark and when we looked at the one

that everyone's currently benchmarking

their systems on that Benchmark had some

issues that were way too hard or

completely impossible to solve leading

the swe bench to systematically

underestimate the model's autonomous

software engineering capabilities I'm

going to say that one more time open I

came out and said that that our testing

indicates that some swe bench talks are

impossible to solve leading to swe bench

systematically underestimating the the

model's autonomous software engineering

capability which basically means that

look we made benchmarks and we realized

that your ones were pretty impossible to

solve and you guys aren't truly

realizing how capable these models truly

are which is an issue because if you've

been paying attention to the space what

you'll know is that there have been

major major updates for example with

fine-tuning you can see that the actual

Improvement since Devon since 4 months

ago this has completely doubled guys and

the software engineering you know bench

you can see right here that since the

Devon area which was around 133% you can

see that that was around here during

that time in 4 months performance has

doubled okay we've had numerous

competitors come out of the works we've

had Amazon's Q developer agent get

38.8% we've got the factory code Droid

that are you know aiming to build you

know like an army of autonomous software

agents and then we of course recently

had cosign Genie which is now the

state-of-the-art model at 43. 8% on that

same Benchmark that Devon was on that

17% Benchmark so for those you know who

are saying that you know this one

doesn't work and this you know is awful

yada y y I would love to see what people

have to say about the actual

improvements because many people are now

dismissing these claims and many people

are now stating that look this thing

isn't good at all this isn't actually

any kind of reliable Improvement but if

we've just been paying attention to the

rate of improvement here we can see that

this is absolutely incredible now I've

made this table in Google and you can

see right here that the improvements

here are rather Stark we can see that

literally at the start of 2024 we were

at 7% you can literally just see right

here 7% and now we are 8 months through

2024 and we at

43.8% and remember Devon was only four

months ago you can see and in the four

months you know we've had you know 38 38

37 36 26 you know things have been

moving rather rather quickly so it's

important to know that now I did some

you know not testing but you know I

spoke to Claude I wanted to ask them how

quickly are we going to be moving

towards 90% because the reason I said

90% is because to go from like 40% to

90% is rather easy and that is going to

be an acceleratory period but to get

from 90% to like 99% is a lot harder

like making gains when you're already

you know 90% of the way there it becomes

you know exponentially harder to make

those additional gains so it says here

that there is a significant jump from

the older models below 10% to the newer

ones about 20% the Improvement seems to

be accelerating with more recent models

showing larger gains this is for many

factors including the fact that you know

all these com companies are now coming

out of stealth showing you know everyone

what they've been building and it says

here that given these observations we

can make a rough estimate the

improvement from the best 2023 model

which is retrievable augmented

generation and Claw 3 Opus to the best

2024 model is about a

36.8% % increase in roughly 8 months

this suggests an average Improvement

rate of about 4.6 percentage points per

month to get from 43.8% to 90% we need

an additional 46.2 percentage points at

the current rate of improvement it would

have take approximately just 10 months

to reach 90% however technological

progress often follows an S curve where

improvements accelerate for a while then

slow down as we approach theoretical

limits and we're likely in the

acceleration phase now considering this

and the rapid re progress I would

estimate that reaching 90% could be

possible with 6 to 12 months from the

latest data in the date placing the

prediction sometime between February and

August

2025 so with Claude 3.5 doing the

analysis here you can see that if we

actually look at the data and we you

know extrapolate out further and say

okay you know we know what's going to

happen Claude is basically predicting

that look within 10 months you know it's

going to be 90% of all swe tasks that

are quite possible and then you know

Amazon Cloud Chief is basically saying

that look you know in the 24 months to

come things are going to be rather

different it doesn't seem that crazy

when you actually break it down number

by number and this is some important

things that you do have to pay attention

to especially if you're in this space

now one of the things that most people

do actually forget and I do think that

this prediction isn't overestimated this

isn't like a hype video where I'm like

oh my God you know AGI and you know 10

months or two months this is more of

like you know trying to keep it factual

and grounded literally based on the you

know benchmarks that we've recently seen

but we've seen like 4.6 percentage

Points each month of course some months

are going to be larger some months are

going to be not as good but I think if

you know if we go at the same rate it's

going to be 10 months we can you know

say that okay even if it's not 10 months

even if we extrapolate and add another

year onto that there's going to be huge

amounts of significant developments now

like I said before there's always one

thing that most people forget and this

is where I think there's going to be you

know huge jumps made in AIS coding

capabilities you know in the areas of

these Frontier you know models doing

stuff that just most people didn't even

take into account okay and that's why I

said the recent videos I'm talking about

you know such as this one right here you

know it's saying that debunking the AI

software engineer yada yada yada this is

terrible always take into account the

actual benchmarks of other AI systems

because that is also important and I did

watch this video from n code. where he's

basically explaining that look hype is a

marketing tool and basically stating

that hype is completely out of control

and that this is you know something that

could potentially not get there just yet

but I do think that what this video

doesn't pay attention to is of course

some of the stuff and of course some of

the more recent developments and some of

the other papers where they actually

talk about other coding stuff which I'm

going to get into in a minute most

powerful marketing tool that's ever

existed let's talk Devon AI for a second

I made a video talking about it right

after the announcement I thought the

founders were super smart but also that

I'm not worried about Devon because

these people are clearly fighting an

uphill battle

I don't know a lot about it the

benchmarks say that this is more

effective at software engineering tasks

than the other llms I wonder if that's

because it just puts a few things

together like it has its own

capabilities to like do research like go

on Google brows stack Overflow and run

code and execute code and they just put

those pieces together more more

cohesively than GPT obviously in a short

amount of time they didn't create their

own complex llm I'm pretty sure they're

using one of the existing llms I didn't

closely examine the evidence or anything

like that it seemed kind of obvious to

me a few months later people are

realizing they were fooled by the hype

Devon is extremely overrated at least

for now if you don't believe me maybe

you'll believe this guy he basically

proved that at this point Devon AI is

more useless than a freshman CS student

who's one week into their first

programming course so what I wanted to

talk about you know after you know

viewing ni's video it was definitely an

insightful video and I do think that it

grounds you know a lot more you know

reality in terms of what's actually

happening because I think what we do

need to pay attention to is the reality

of things you know a lot of things have

blown out in that video he talks about

Tesla's been selling for self driving

for years you know companies make claims

all the time you've got FTX you've got

theranos this is something that usually

does happen with tech companies but most

people aren't paying attention to what

these Frontier companies are saying okay

for example um this is a paper that

didn't get a lot of you know information

slash like data you know people just

basically didn't speak about this paper

but essentially this is the alpha code 2

technical report okay and it talks about

how alpha code was the first AI system

to perform at the level of the median

competitor in competitive programming a

difficult reasoning tasks involving

Advanced maths logic and computer

science and this paper introduces Alpha

code 2 a new and enhanced system with

massively improved performance powered

by Gemini Alpha code 2 relies on the

combination of powerful LMS and a

bespoke searching and reranking

reckoners when evaluated on the same

platform as the original Alpha code we

found that Alpha code 2 solved 1.7 times

more problems and performed better than

85% of competition participants now I'm

just going to you know ground this in

you know what you can understand here

for including myself we can see here

that uh basically this was done on the

code forces okay and this was able to

get the 85 percentile on the code forces

so code forces is a wellknown platform

for competitive programming with a

ranking system that categorizes

participants based on their performance

in the contest the ranks include newbie

pupil specialist expert candidate master

master International Master grandmas

International Grandmaster legendary

Grandmaster and reaching 85th percentile

on code forces means performing better

than 85% of all competitors and

according to the data Alpha code 2 ranks

between the expert and candidate Master

levels which were quite Advanced now if

you want to ground that in terms of how

good it could potentially show us how

you know these systems are going to be

in terms of you know coding abilities

and what we can look at for the future

you can see here that for code forces

the expert level which is you know the

top 70% to 85% participants at this

level have a strong understanding of

algorithms and data structures they're

proficient in solving standard

competitive programming problems and are

beginning to solve more challenging

problems that require deeper insights or

more Advanced Techniques and the

comparison to software engineering roles

here basically says that typically

focusing on learning and applying

basically programming Concepts writing

clean code contributing to projects

under supervision the main problem

solving required for competitive

programming at the expert or candidate

level is generally beyond what is

expected of a junior software engineer

and this is usually what you get at a

midlevel to senior level software

engineer and engineers at this level are

expected to have strong problemsolving

skills understanding algorithms and data

structures and be capable of Designing

efficient and scalable solutions they

might not participate in competitive

programming but they should be

comfortable with similar challenges when

needed the point here guys is that what

we have is a scenario where you know we

should discount the technology as for

what we've seen recently as that every

single month this is going to you know

continually increase one of the things

that I think most people aren't paying

attention to is the fact that you know

even open AI have openly said that this

is going to be something themselves that

they are going to be tackling for

example take a look at this clip of

samman actually talking about what one

of the biggest areas of improvement is

likely going to be for him moving

forward more to gain there graph a

couple of decades in the future be like

H something changed yeah are there

application

or areas you think are most promising in

the next 12 months I'm sure I'm biased

just because of where what we do here

but Cod I think is a it's a really big

one and the only reason I include that

clip is because opening eye have

previously always surprised us with

their capabilities and you know they

tend to surprise us in areas that we we

aren't even pretty much focusing on so

the takeaways are simple guys what we

have here is a situation where you know

over the next 24 to 48 months we could

see a complete shift in terms of the

main role of software Engineers I still

think that we're going to struggle to

get from 90% all the way to you know 99%

so I think that there will be like this

scurve growth where you do have a

situation where right now you know if

you look at the benchmarks we are

currently during that acceleration

period where you know you've got these

systems you know at the start it's

really hard to make progress then all of

a sudden the jumps happen the jumps

happen then as you get to like 90 95%

that's where the taper off is so I do

think that if you're like a senior level

software engineer to someone that's

really cracked I do think that you know

your job is probably not going to change

for the most part because you're still

going to have to understand how all of

those systems work together and some of

the most difficult tasks but I do think

that you know as time goes on you know

AI is probably going to eat like the

bottom area where you're going to be

writing code and you know as time goes

on it's going to continue continue to

continue to continue to you know shift

the role of you know many software

Engineers I think one of the most

interesting things from all of this is

how companies are going to change in

terms of their hiring process is someone

just going to be you know managing AI

agents is it going to be a new you know

AI agent structure that allows you to

you know manage the entire code base

you've got AI systems that are going to

be working with millions and millions of

context lengths either way I think this

is something that you know is going to

really really change in the next two to

four years arguably one of the biggest

changes that's going to happen but

either way I don't think it's all doom

and gloom I just think that the role

will change and I think it's going to be

fascinating to see how that happens if

you did enjoy the video don't forget to

leave a like do forget to subscribe and

I'll see you guys in the next one