Is AI Replacing Software Engineering?

Summary

The speaker explores the question of AI replacing software engineering, presenting data on AI's current capabilities in solving coding issues and emphasizing the limitations. They compare AI to self-driving technology, highlighting that while impressive, AI lacks human-like logical thinking. The speaker discusses the impact of AI on software engineering jobs, suggesting a balance between productivity gains and increased software creation. They offer advice for individuals in the industry, recommending familiarity with AI models, effective prompting techniques, and understanding the diverse tools available to enhance their work.

Takeaways

• 🧠 AI as a software engineer is still in its early stages, with systems like Devon, Factory Code Droid, and AER solving only a small percentage of real-world software issues.
• 🔍 The success rates of AI in solving issues are impressive but may not reflect the complexity or randomness of the issues, as they tend to be on the easier side.
• 📈 AI's current capabilities are compared to self-driving technology, where it can perform tasks autonomously but is not yet at a level to fully replace human drivers or engineers.
• 🤖 AI lacks human-like logical thinking and understanding of intentions, which means it requires human guidance to achieve desired outcomes.
• 📊 The speaker used a self-driving analogy to illustrate that while AI can perform certain tasks, it is not yet advanced enough to replace the need for human software engineers.
• 📈 The number of software engineering jobs and the availability of engineers are influenced by various factors, including AI, but primarily by economic conditions like interest rates.
• 💡 There is a debate on whether AI will increase productivity, reducing the need for engineers, or lower the cost of software creation, leading to more jobs.
• 📉 The job market for junior software engineers has been tough, with fewer job postings and a perception that AI tools might be a threat to their employment.
• 🛠️ For those in or entering the software engineering field, it's important to be familiar with different AI models and understand their capabilities.
• 🗣️ Effective prompting is key when working with AI; clarity of intention, context, and detailed instructions can improve AI's performance.
• 🧰 Software engineers should view AI as an additional tool in their toolkit, to be used appropriately alongside traditional tools for different tasks.

Q & A

What is the main topic of the talk?

The main topic of the talk is whether AI is replacing software engineering.

What is Devon and what claim does it make in the context of AI software engineering?

Devon is an AI system that claims to be the first AI software engineer. It claims to have autonomously solved about 14% of real-world software engineering issues on a dataset called Sweet Bench.

What are Factory Code Droid and AER, and what percentage of issues did they solve?

Factory Code Droid and AER are other AI systems that claim to have solved about 19% of the issues on the same dataset as Devon.

What are some limitations of the AI systems mentioned in the script when solving software engineering issues?

The limitations include that the issues solved by these AI systems are not necessarily random and are more likely to be on the easier side. Also, it doesn't indicate the quality of the solutions, only that they passed the tests.

How does the speaker compare AI in software engineering to self-driving technology?

The speaker compares AI in software engineering to self-driving technology by stating that just as a self-driving system that can autonomously drive on 20% of public roads is impressive but not enough to replace human drivers, AI in software engineering has made progress but is not yet ready to replace human engineers.

What is the speaker's personal view on AI's current capabilities in logical thinking and understanding human desires?

The speaker believes that AI, as it is currently developed, does not think like a human. While it is intelligent in certain tasks, it lacks the same logical thinking capabilities and understanding of human desires.

What is the event called 'AI Dev Tools Night' and what was the speaker's role in it?

AI Dev Tools Night is an event hosted by the speaker in San Francisco. The speaker used AI to analyze responses from a survey during the registration process for the event.

What was the issue the speaker faced when using AI to visualize survey responses, and how did they resolve it?

The issue was that the AI categorized each slightly different response as a different category instead of grouping similar responses. The speaker resolved it by providing more detailed instructions and context to the AI, and manually guiding it to produce the desired results.

What is the current state of the job market for software engineers according to the script?

According to the script, the job market for software engineers has become more challenging with fewer jobs available compared to the number of software engineers seeking employment.

What are the two contrasting views on the impact of AI on software engineering jobs?

One view is that AI will increase the productivity of software engineers by 20-30%, reducing the number of engineers needed. The other view is that AI will lower the cost of creating software, leading to more software being created and thus more software engineering jobs.

What advice does the speaker give to individuals considering a career in software engineering?

The speaker advises individuals to be familiar with different AI models available in the market, learn effective prompting techniques, and understand the different types of development tools that can be used in their work.

What is the speaker's prediction for the future of the software engineering job market?

The speaker predicts that in a few years, the market will be slightly better with more software engineering jobs, based on current trends and the potential effects of AI.

What is the role of Sourcegraph Code in the context of the talk?

Sourcegraph Code is an open-source coding assistant that the speaker used as an example to demonstrate the current capabilities and limitations of AI in software engineering. The speaker also works at the company behind Sourcegraph Code.

Outlines

00:00

🤖 AI's Role in Software Engineering

The video script discusses the question of AI replacing software engineering, starting with an introduction to AI tools like Devon, Factory Code Droid, and AER, which claim to solve a percentage of real-world software engineering issues. The speaker emphasizes that while these AI systems can solve a portion of the issues, they are more likely to tackle the easier ones and do not necessarily reflect the quality of the solutions. The analogy of AI's current capabilities to self-driving systems that can only autonomously drive on a limited percentage of roads is used to highlight the limitations of AI in software engineering. The speaker also points out that AI lacks the logical thinking and desires of humans, suggesting that AI is a tool to be used in conjunction with human expertise rather than a replacement for it.

05:00

📊 AI's Impact on Software Engineering Jobs

This section of the script delves into the impact of AI on software engineering jobs, using the analogy of market demand and supply to visualize the situation. The speaker discusses the potential for AI to increase productivity, which could lead to fewer software engineers being needed, but also the possibility that reduced costs due to AI could lead to more software being created, thus increasing the demand for software engineers. The speaker provides data points such as the increase in hiring postings and optimism among hiring managers, suggesting that the market may improve in the future. However, the speaker also acknowledges the challenges faced by junior engineers and the importance of considering passion and potential contributions to the industry when deciding to pursue a career in software engineering.

10:01

🛠️ Navigating the AI-Enhanced Software Engineering Landscape

The speaker offers advice for individuals in the software engineering field regarding AI integration. The first piece of advice is to familiarize oneself with different AI models and their capabilities, using leaderboards like lmis.org as a reference. The second is to learn effective prompting techniques to get the most out of AI tools, emphasizing the importance of clear intentions, context provision, and detailed prompts. The third piece of advice is to understand the various types of development tools available, from auto-completion systems to AI coding agents, and to use the right tool for the job at the right time. The speaker highlights the importance of integrating AI tools into one's toolkit and using them effectively to enhance productivity and outcomes in software engineering.

15:02

🌐 Embracing AI as Part of the Software Engineering Toolkit

In the final paragraph, the speaker reiterates the notion of AI as an additional tool in a software engineer's toolkit, rather than a replacement for existing tools. The speaker encourages software engineers to continue learning and adapting to new tools, including AI, to stay relevant and effective in their roles. The speaker also discloses their affiliation with Sourcegraph, a company actively working on AI development tools, and encourages the audience to explore the company's offerings. The script concludes with a thank you to the audience, followed by applause.

Keywords

💡AI replacing software engineering

This phrase is central to the video's theme, discussing the potential for artificial intelligence to supplant human labor in the field of software engineering. The video explores the capabilities and limitations of AI in solving programming issues and contributing to the development process. For instance, it mentions AI systems like Devon and Factory code Droid, which claim to autonomously solve a percentage of real-world software issues, thus raising the question of AI's role in the future of software engineering.

💡Devon

Devon is referred to as an AI software engineer in the script, highlighting a specific AI system that claims to be capable of addressing software engineering tasks. The video uses Devon as an example to discuss the extent to which AI can autonomously solve issues from a dataset of real-world software engineering problems, emphasizing the ongoing debate about the capabilities of AI in this domain.

💡Factory code Droid

Factory code Droid is another AI system mentioned in the script, which, like Devon, is purported to solve a certain percentage of software engineering issues. The script uses this example to compare the performance of different AI systems in tackling real-world programming challenges, thereby contributing to the broader discussion about the effectiveness and potential of AI in software engineering.

💡AER

AER is briefly mentioned alongside Devon and Factory code Droid as an AI system that also claims to solve a percentage of software engineering issues. While not elaborated upon extensively in the script, AER serves to illustrate the competitive landscape of AI tools striving to automate aspects of software development.

💡Quality of PRS and contributions

PRS, or Pull Requests, are a feature of version control systems that allow developers to propose changes to a project. The script points out that while AI can pass tests, there is a question about the quality of the contributions it makes, suggesting that the ability to pass tests does not necessarily equate to making meaningful, high-quality contributions to software projects.

💡Self-driving analogy

The video uses the self-driving analogy to illustrate the current state of AI in software engineering. It suggests that just as a self-driving system that can only navigate 20% of public roads is not yet ready to replace human drivers, AI's current capabilities in software engineering are impressive but not sufficient to replace human engineers entirely.

💡AI Dev tools night

AI Dev tools night is an event mentioned in the script that the speaker is hosting. It serves as a concrete example of how AI is being integrated into the software engineering community, with the event focusing on AI's role in development tools. The speaker uses this event to discuss the practical application of AI in analyzing survey responses, further exploring AI's capabilities and limitations.

💡Sourcegraph Code

Sourcegraph Code is an open-source coding assistant that the speaker mentions in the context of the AI Dev tools night event. The script uses this tool to demonstrate how AI can be used to analyze survey responses and generate visualizations, highlighting both the potential and the challenges of using AI in practical software engineering tasks.

💡Chat-oriented programming (CHOP)

CHOP is a concept introduced in the script that refers to a programming paradigm where developers interact with AI systems through conversational interfaces. The video suggests that as AI becomes more integrated into the development process, the traditional line-by-line coding may matter less, and developers will need to adapt to this new way of working with AI.

💡Software engineering jobs

The script discusses the impact of AI on the availability of software engineering jobs, using a visual representation to illustrate the balance between the number of available jobs and the number of software engineers. It raises the question of whether AI will lead to a decrease in job opportunities or whether the cost reduction in software creation will lead to an increase in software engineering jobs.

💡Interest rates

The video mentions interest rates as a significant economic factor that may influence the number of software engineering jobs available. It suggests a correlation between rising interest rates and a decrease in software engineering job opportunities, indicating that economic conditions play a crucial role in the job market, alongside the impact of AI.

💡Hacker News

Hacker News is a social news website focusing on computer science and entrepreneurship. In the script, it is used as a source to illustrate trends in job postings for software engineers, suggesting that there is a record number of hiring postings, which may indicate a positive job market despite the challenges posed by AI.

💡Junior software engineers

The script discusses the impact of AI on junior software engineers, noting that the number of jobs for juniors has been decreasing over the years. It suggests that newcomers to the field may face greater challenges due to the integration of AI in software development, affecting their job prospects and the need to adapt to new tools and methodologies.

💡Large language models

Large language models are a type of AI system that can process and generate human-like text based on input. The script advises software engineers to familiarize themselves with different models available in the market, understanding their strengths and weaknesses, as these models are becoming increasingly important tools in the software development process.

💡Effective prompting

Effective prompting refers to the technique of providing clear and detailed instructions to AI systems to elicit the desired response or output. The script uses the example of using AI to analyze survey responses, emphasizing the importance of clear intentions, context provision, and detailed prompts to achieve satisfactory results with AI tools.

💡Different types of Dev tools

The script discusses the evolution of development tools, from autocomplete systems to chat systems and AI coding agents. It suggests that software engineers should understand the various tools at their disposal, including AI tools, and learn to use the right tool for the right job, adapting to the changing landscape of software development.

Highlights

AI's potential to replace software engineering is being questioned, with current AI systems like Devon, Factory code Droid, and AER solving only a fraction of real-world software issues.

AI systems have shown the ability to autonomously solve about 14% to 19% of issues in open-source Python projects, but the quality and randomness of these solutions are debatable.

The analogy of AI in software engineering to self-driving cars is made, suggesting that while impressive, AI is not yet at the level to fully replace human engineers.

AI's current limitations include a lack of human-like logical thinking and understanding of human desires, which are fundamental to software engineering.

A concrete example demonstrates the need for human guidance in using AI for data analysis and visualization, highlighting AI's inability to autonomously understand and categorize data effectively.

The speaker's experience with AI in event registration analysis shows that despite AI's assistance, significant human input was required to achieve satisfactory results.

AI's role in coding is compared to the evolution of development tools, suggesting a shift towards 'chat-oriented programming' where coding by conversation becomes more prevalent.

The impact of AI on software engineering jobs is discussed, with a visual representation of job availability versus the number of engineers, indicating a challenging market.

Interest rates are identified as a significant factor affecting the number of software engineering jobs, potentially more impactful than AI.

Two opposing views on AI's effect on jobs are presented: one suggesting increased productivity and fewer engineers needed, the other predicting more software creation and job growth due to lowered costs.

Optimism is expressed for the long-term creation of more software engineering jobs, despite the uncertainty of AI's exact impact.

Current market trends and hiring manager optimism suggest a potentially improving job market for software engineers in the near future.

A tough market for junior software engineers is highlighted, with a decline in job postings and concerns about AI's threat to their jobs.

Passion and the desire to excel in the field are encouraged as the driving factors for entering or staying in software engineering, despite market challenges.

Three pieces of advice for individuals in or entering the software engineering industry are given: familiarize with AI models, learn effective prompting, and understand the variety of dev tools available.

The importance of understanding and utilizing the right tool for the right job in software engineering is emphasized, including the integration of AI tools into the developer's toolkit.

Sourcegraph Cod is mentioned as a company actively working on AI tools for software engineering, with the speaker disclosing their affiliation.

Transcripts

this is the question we're going to

cover in this talk is AI replacing

software engineering you know I'm going

to show you a lot of data a lot of

charts I'm GNA try to Define what this

question even means in the first place

so we'll get started first of all the

first piece of tech that will come to a

lot of people's minds when I ask this

question is called Devon Devon they

claim to be the first AI software

engineer and the claim is highly you

know arguable and debatable but at least

on this particular data set called sweet

bench that's a data set of real world

soft engineer issues on open source

python projects they were able to show

that their fully autonomous AI software

engineering system was able to solve

about 14% of those GitHub issues and

then later on this other one called

Factory code Droid came about and they

said they were able to solve about 19%

of those issues this other one called

AER said they were able to solve about

19% of those issues as

well and when you look at these numbers

you might say that looks pretty

impressive and it is but there are a

couple things to keep in mind one is

that there are not necessarily random

19% of those issues not completely Rand

random they're more likely to be on the

easier side of the issues both for

humans and Ai and then the other thing

is this doesn't say anything about the

quality of the PRS and the

contributions it just says that they

were able to pass the tests and that's

pretty much

it but still you might say you know with

this strong kind of improvement

is AI replacing software

engineering the way I like to personally

think about it is kind of like

self-driving you know if a company came

about and said we developed a

self-driving system that's able to drive

autonomously on 20% of all public rows I

would say okay that's that's pretty

impressive but that's not quite as good

as it would need to be to be able to

replace driving completely in that

particular anal ology and same thing

with software engineering and the

fundamental problem to me is that AI the

way it's currently developed it doesn't

think like a human does you know it is

really intelligent at certain tasks it

is able to solve certain types of

tasks but it doesn't have the same

logical thinking capabilities that

humans do it doesn't always know what we

want it doesn't have a desire like

people do so it's fun fundamentally

different and that's really where you

know we need to come in as humans and

kind of work with AI to get the results

we want and I wanted to show you a

concrete example of that here and that

is this event I'm hosting in San

Francisco in a few days I'm actually you

know flying out there tomorrow for that

it's called AI Dev tools night and for

this event I asked a few questions

during the registration process I said

how did you hear about this Meetup and

would you like a chance to try this open

source coding assistant Source graph Cod

and I wanted to use AI to analyze these

responses basically I wanted to use AI

to write python to visualize these

responses so I went ahead and downloaded

a CSV version of the survey responses

and then I cleaned it up I anonymized of

course very important and then I

uploaded it to

chbt and I said can you make charts out

of these survey

responses and the result I got looked

like this so it might be kind of hard to

see but on the right it's not too bad it

is showing a graph of you know how many

people said yes versus how many people

said no but on the left what it's doing

is it's trying to categorize

each slightly different response to this

free form text question as a different

category but that's not what it should

do you know what it really should do is

it should kind of manually inspect all

the responses and then categorize them

sort of manually using python but it

didn't know to do that like I said AI

sometimes doesn't know what we want as

people so that's where I needed to come

in and go back to the beginning of the

conversation and I needed to kind of go

step by step I needed to first ask do

you understand this data in the first

place you know yes or no and I needed to

give it the same data in a textual

format you know it's the same CSV data

but I needed to copy and paste it in a

textual format and then after that I

needed to ask it to make pip charts out

of these you know while grouping similar

categories and ignoring non responses

and any needed to give it a lot of

details like that and then on top of all

of that I needed to upload that CSV file

again and then I need to say remember

this this data is the same as the

textual format data I gave you you just

need to you know Define the

categorization and grouping with

python so I needed to go through a lot

of work you know it took me a

significant amount of work to go through

that and then finally I was able to get

it to produce python code that produced

satisfying

results so here you might ask did AI

replace software engineering or coding

whatever you want to call this

particular to

example and in a way it

did because I didn't have to write the

underlying python code you know AI did

it for me but at the same time I needed

to go in as a human software engineer

you know with some knowledge of python

data

visualization the service structure and

all of that to kind of instruct it to do

exactly what I wanted you know I needed

to really work with it so AI as it

currently stands it's not ready to

replace software engineering on its own

completely but we are moving more into

this world where line by line coding

matters less

and you know coding by chat or you know

coding with AI matters more you know we

need to work with AI and some people

call that chat oriented programming or

chop for short so to me a more relevant

question here is is AI replacing

software engineering

jobs and I like to put this question in

perspective by visualizing it in this

way so in this chart the area above the

line represents the number of software

engineer jobs that are available in that

particular year and the area inside the

circle represents the number of software

Engineers that are available let's say

in the world and as you probably know in

2020 2021 you know the market was much

better for soft Engineers more soft

engineer jobs so it was much easier for

a typical software engineer to get a job

keep a job get a raise but this year you

know it's much harder fewer soft

engineer jobs and it's hard for typical

engineer to do all of those three things

and the question you might ask here is

in a few years is it going to look like

this where the market is even harder or

is it going to look like this where the

market is slightly easier with more

software engineering

jobs this is a tough question to answer

partly because AI is not the only factor

that goes into this you know it's only

one of the factors and it's probably not

even the main factor in my opinion one

of the main factors that goes into it is

actually the interest ratees so for the

P past few years the interest rate has

gone

up and at the same time the number of

soft engineering jobs has gone down so

there's a you know seemingly strongly

strong correlation there but you might

say Okay what is

ai's effect in particular on jobs though

there are kind of two sides to this

arguments one side says each software

engineer is going to be more productive

you know from what I've seen 20 to 30%

more productive and therefore we'll need

fewer software

Engineers but the other side says the

cost of creating software is going to go

down because of AI and therefore more

software will be created because you

know when something's cost go goes down

more of it will be created in

general and for that reason you know

there would be be more soft engineering

jobs and there is a merit to both of

these

arguments it's impossible to know for

anyone you know which side is going to

win out over the long term but I'm

personally slightly more optimistic

towards more software jobs being created

over the long term and if you look at

the present and the you know immediate

future there are a couple of positive

signs as well so this one is from May of

this year that shows there were sort of

a record number 12 month High number of

who is hiring postings on Hacker News in

particular and then this one is a news

article that says 81% of hiring managers

are optimistic about hiring plant for

the rest of the year so when you combine

all of these different factors facts on

the ground and different stories what

people are experiencing plus potential

effects of AI my best educated guess is

that in a few years Market is going to

be slightly better there will be more

soft engineer jobs that's my best

prediction you know I could be wrong but

that's where I am with this and this is

the question I get a lot so I need to

adjust this should I go into softw

engineering if someone had asked me

should I go into software engineering

just to make money in 2016 I would have

said yes I mean it's it's a nice way to

make make money just do that but this

year I would say be careful about it and

part of the reason is because it's been

a tough market for juniors in particular

so this is a graph that shows the number

of jobs for different

levels on Hacker News again and if you

look at the blue lines

for juniors both in absolute numbers and

percentages the number of jobs for

juniors has been going down over the

past few years and I think Juniors

themselves you know understand this

Dynamic to some extent if you look at

stock overflow 2024 survey you know you

find this question are AI tools a threat

to your job and people who were learning

to code were more likely to say yes are

they they are a threat to my job than

professional developers at the same time

you know this still a good industry in

my opinion if you're like really

passionate about it whatever it is right

like not necessarily software

engineering you know whether it's QA or

design or whatever that might be if

you're passionate about it if you want

to make a lot of contribution to it if

you feel like you can be one of the best

then definitely go for it I would

encourage it but if not be careful you

know think about it carefully and the

final question I'm going to try to

address in this talk is what can I do as

an individual if I already decided to go

into the industry or if I'm already in

the industry some of you are you know

maybe in that

spot in that case I have three pieces of

advice that I recommend the first one is

to be familiar with different models

that are available in the market you

know different large language models

currently and understand you know to

some extent you don't have to be an

expert but you want to understand to

some extent what they're good at which

ones are you know better than which

other ones roughly speaking currently

you know if you look at this uh popular

leaderboard lmis org you see that Gemini

1.5 Pro is the most advanced model

according to their metrics but that's

for the overall category if you look at

the coding category you find that claw

3.5 sunet is the most advanced model

according to the metrics and it has been

regarded as the most advanced model for

coding in general as well so I would say

learn about you know this different

models what they're good at and

basically find a tool that fits your job

the second piece of advice I would give

is learn effective prompting I think the

example I gave earlier is a good example

of that it comes down to three things in

my opinion you want to make your

intention really really clear you want

to make your context clear provide as

much context as possible maybe a lot of

copy and pasting and then you want to

provide a lot of details in your prompt

and combining all of these if that's not

even enough then you might need to go

through a lot of trial and error to see

what works whether it's back and forth

conversations or you know trying out

different prompts and then the third

thing I would keep in mind is that there

are different types of Dev tools that

you can use as a soft engineer or coder

in general you know we started with

really you know Auto completion code

completion systems you know kind of like

co-pilot we moved on to you know chat

systems chat gbt other ones Claud more

advanced chat systems and then we also

now have ai coding agents I think the

way I like to see is that you know part

of your job as a soft engineer has

always been to you know understand

what's in your tool set you know is it

Google is it your code editor is it

something else is it stack

Overflow those tools have always been in

your tool set what's changed with AI is

simply that you have more Tools in your

tool

set you know not any single one of these

tool is going to replace all the other

ones like you know AI coding agents are

not going to replace the chat systems

completely and chat systems are not

going to replace the autocomplete

systems

completely so you want to understand

that there are different tools available

in your tool set now additional tools

and as it's always been the job as part

of your job as a softare engineer you

want to learn to use the right tool for

the right job for at the right time I

hope that makes sense and I have to say

one tool and Company that's been working

on all of these surface areas actively

is called Source graph Cod it's company

I work at now you know for full

disclosure and I even gave a talk about

it in San Francisco last time I was

there so feel free to check it and thank

you so much everyone

[Applause]