Data Engineers = 80% Software Engineers

Summary

This video explores the evolving skill set expectations in the data engineering job market from 2024 onwards. It highlights the increasing demand for software engineering skills like API services and basic web development, in addition to traditional data engineering competencies. The speaker emphasizes the importance of adapting to these changes to enhance job prospects and stay relevant in the industry.

Takeaways

• 😀 The video discusses the evolving skill set expectations in the data engineering job market, particularly from 2024 onwards.
• 🔍 Predictions from 2020-2023 suggested that AI and data science skills would be crucial for data engineers, but current trends indicate a shift.
• 💼 The job market now expects data engineers to have a broader set of skills, including software engineering competencies.
• 👨‍💻 Primary skills of software engineers include programming languages, SQL, data structures and algorithms, databases, version control, software development life cycle, web development, cloud computing, testing, debugging, and mobile development.
• 🤔 Data engineers typically already possess many of these skills, such as programming languages, data structures and algorithms, databases, version control, DevOps, operating systems (Linux), and cloud computing.
• 🚀 However, data engineers may lack skills in API services and basic web development, which are becoming increasingly important.
• 🏢 Companies are building internal products for developers, which often require basic web UIs and API services, leading to a demand for these skills in data engineers.
• 💡 The video suggests that while these additional skills are not a major requirement, having them can be an advantage and can increase job opportunities.
• 📈 The impact of not knowing these additional skills in interviews is minimal, as the primary focus remains on core data engineering skills.
• 📚 Learning new skills like basic web development and API services can be done with minimal effort and can provide an edge in the job market.
• 🌐 The speaker encourages viewers to stay updated with new skills and technologies, as the tech industry is constantly evolving.

Q & A

What is the main focus of the video?

The video focuses on the evolving skill set requirements for data engineers, particularly highlighting the increasing expectation of software engineering skills in data engineering roles.

What was the prediction for the data engineering job market in 2020-2025?

The prediction suggested that people should learn AI and data science and add these to their data engineering skill set to increase job opportunities.

What has changed in the data engineering job market from 2024 onwards?

From 2024, the data engineering job market has started to expect more software engineering skills as part of the job description (JD).

What are some primary skills of software engineers mentioned in the video?

The primary skills mentioned include programming languages, SQL, data structures and algorithms, databases, version control systems, software development life cycle, methodologies like Agile and Scrum, web development, cloud computing, testing and debugging, and operating systems like Linux.

Which skills do data engineers typically already possess according to the video?

Data engineers typically already possess skills in programming languages, data structures and algorithms, databases (especially SQL), version control systems, DevOps (including Kubernetes and Docker), operating systems (like Linux), software development methodologies (like Agile and Scrum), and cloud computing.

What skills are data engineers currently lacking according to the video?

Data engineers are currently lacking skills in API services and web development, as these are not typically part of their skill set.

Why are companies expecting data engineers to know API services and basic web development?

Companies are building internal products for developers, which often require a basic UI for user interaction. Data engineers are expected to have some knowledge in these areas to contribute to these projects without the need for hiring additional software engineers or frontend developers.

How does the video suggest data engineers can acquire knowledge in API services and web development?

The video suggests that spending a few hours on the internet and YouTube can help data engineers learn the basics of API services and web development.

What impact does not knowing HTML, CSS, or API services have on a data engineer's interview?

Not knowing these skills does not significantly impact the interview outcome. Interviewers are primarily interested in the candidate's data engineering skills, but having some knowledge in these areas can be an added advantage.

What is the advice given to data engineers regarding learning new skills in the video?

The advice is to be open to learning new skills, such as API services and basic web development, as this can increase visibility and opportunities in the job market. It is also beneficial for cost-saving within companies.

Outlines

00:00

💼 Evolving Skill Sets in Data Engineering

This paragraph discusses the changing landscape of data engineering job market and the evolving skill set expectations. Initially, it highlights the importance of AI and data science in the past, but shifts focus to the current trend where software engineering skills are increasingly being integrated into data engineering roles. The speaker emphasizes the need for data engineers to be proficient in programming languages, SQL, data structures, algorithms, databases, version control systems, software development life cycles, methodologies like Agile and Scrum, web development, cloud computing, testing, debugging, and operating systems like Linux. It also points out that while data engineers are expected to have some knowledge of API services and web development, these are not core requirements but rather additional skills that can enhance their profile. The speaker concludes by stating that the primary focus should remain on data engineering, but having a broader skill set can be advantageous.

05:01

🏢 Internal Product Development and Its Impact on Job Market

The second paragraph delves into how companies are building internal products for developers, which are essentially tools and frameworks that facilitate data movement, replication, and management. These products are designed to be user-friendly for developers within the organization, reducing the need for external hiring for specific tasks like front-end development or API services. The speaker explains that while these skills are not mandatory, having a basic understanding of HTML, CSS, and web development can be beneficial. The paragraph also touches on the interview process, where these additional skills are considered but not critical. The speaker suggests that even if a candidate lacks these skills, their ability to learn and adapt is more important. The emphasis is on the cost-effectiveness of having multi-skilled employees and the potential for data engineers to contribute to internal product development, thereby increasing their value within the company.

Keywords

💡Data Engineering

Data Engineering is the practice of designing, building, and maintaining the systems and processes that enable the collection, storage, and transformation of data. In the video, it is highlighted as a field where job market expectations are evolving, requiring data engineers to also possess skills traditionally associated with software engineering.

💡Software Engineering

Software Engineering refers to the systematic approach to the design, development, and maintenance of software. The video emphasizes that data engineers are increasingly expected to have a software engineering skill set, which includes knowledge of programming languages, databases, and software development methodologies.

💡AI and Data Science

Artificial Intelligence (AI) and Data Science are fields that involve using algorithms, statistical models, and machine learning techniques to analyze and interpret complex data sets. The script mentions that these areas were previously emphasized as essential for data engineers, but the current trend is shifting towards software engineering skills.

💡Job Market Predictions

Job Market Predictions refer to forecasts about future trends in employment opportunities and skill requirements. The video discusses how predictions for the data engineering field have changed, moving from a focus on AI and Data Science to a greater emphasis on software engineering skills.

💡Programming Languages

Programming Languages are formal languages designed to communicate instructions to a computer. The script notes that knowledge of programming languages is a primary skill set for software engineers and is increasingly expected in data engineers as well.

💡SQL

SQL (Structured Query Language) is a standard language for managing and manipulating relational databases. The video highlights SQL as a key skill that data engineers already possess, which is also relevant to software engineering.

💡Version Control Systems

Version Control Systems are tools that help software teams manage changes to source code over time. The script mentions that familiarity with version control systems like Git or GitHub is a common requirement for both software and data engineers.

💡DevOps

DevOps is a set of practices that combines software development (Dev) and IT operations (Ops) to shorten the system development life cycle and provide continuous delivery of high-quality software. The video discusses how DevOps skills, including knowledge of Kubernetes and Docker, are relevant in the data engineering field.

💡API Services

API (Application Programming Interface) Services are sets of protocols and tools for building software applications. The script points out that data engineers are now expected to understand how to create and work with APIs, which is traditionally a software engineering skill.

💡Web Development

Web Development involves creating and maintaining websites, typically using languages like HTML, CSS, and JavaScript. The video suggests that basic web development skills are becoming more important for data engineers, especially in the context of building internal tools and products.

💡Cloud Computing

Cloud Computing is the delivery of computing services, including servers, storage, databases, networking, software, analytics, and intelligence, over the Internet. The script mentions that working with cloud platforms like Microsoft Azure and Google Cloud is increasingly relevant for data engineers.

Highlights

Data engineers and software engineers are increasingly expected to have overlapping skill sets.

In 2022 and 2023, the job market predicted a need for AI and data science skills in data engineering roles.

From 2024 onwards, the data engineering job market is expecting additional skills beyond AI and data science.

Software engineering skills are now part of the expectations in data engineering job descriptions.

Primary skills of software engineers include programming languages, SQL, data structures, algorithms, databases, version control, and software development methodologies.

Data engineers typically already possess skills in programming languages, data structures, algorithms, databases, and version control systems.

DevOps skills, including Kubernetes, Docker, and workflow schedulers like Airflow, are also commonly known among data engineers.

Linux operating system proficiency is a must-have for data engineers.

Cloud computing skills are increasingly being integrated into data engineers' skill sets.

API services and web development are areas where data engineers might lack expertise and are now being sought after in the job market.

Understanding and creating APIs is becoming crucial for data engineers as companies build internal developer-focused products.

Basic web development skills, such as HTML, CSS, and JavaScript, are being expected for building internal tools and UIs.

Mobile development is not a focus area even for software engineers currently, and can be excluded from the essential skills list for data engineers.

Companies are building products for internal use by developers, such as data movement and replication frameworks.

Data engineers are expected to have some knowledge of front-end development to contribute to these internal product developments.

Interviewers are considering knowledge of software engineering skills as 10-20% of the evaluation criteria for data engineering roles.

Even without prior knowledge of certain software engineering skills, data engineers can still be considered if they show a willingness to learn.

Learning additional skills like API services and basic web development can be an add-on and provide an advantage in the job market.

Data engineers are encouraged to stay updated with new technologies and integrate them into their skill sets for better job opportunities.

The speaker has an Instagram page called 'the data Tech' where they upload useful short videos related to data technology.

Transcripts

hi welcome to the data Tech 100% of data

Engineers or 80% of software Engineers

100% of software Engineers are just 50%

of data engineer okay what is this all

about you're saying something like a

calculation okay so this video is going

to be completely an interesting fact so

please do watch this

completely so roughly in 2022 and

2023 the prediction of data engineering

job market and work Market in 2020 25

and 2026 was like okay people should

learn uh AI data science and they have

to add this as part of their data

engineering skill set so then only like

we will be getting lot of job

opportunities and data engineering but

what happening now because always the

prediction may it change so what is

happening now from the beginning of 2024

right the data engineering job market

have adding few more skill set as part

of the expectation in the JD so what is

that is that AI data science no it's not

like it's not that

actually so recently so I do have many

contacts in my LinkedIn and people used

to share their interview experience with

me and many of my friends and my ex-

colleagues have attended lot of

interviews in the recent times so this

is not just within this month it's it

happened I I've been I have been doing

this research for almost last three

months with my connections and what I

come to know is people are expecting

software engineering skill set as part

of the data engineering skill skill set

okay so now as a data engineer how

should I know whether what are all the

skill sets or software engineering and

whether do I already know that or not so

that's why I started with 100% data

Engineers or 80% software Engineers

already okay so what are the primary

skill set of software Engineers okay

let's start doing the list programming

language and SQL and then you need to

know data structures andal algorithms

and then databases Version Control

System Dev apps software development

life cycle and methodology like you need

to know the zajil methodology scrum and

all those stuff web development cloud

computing testing and debugging so you

need to know API services and how to

create an API how to write an API in any

any coding it could be on what is API in

first place and then you need to know

operating system especially Linux and

last mobile development so these are all

the some of the primary skill set for

software engineer a typical software an

average software engineer should know

all this as part of their skill okay so

now as a data engineer what I know

already from this list is very maximum

right if you see we know programming

languages and we know data structures

and algorithms right and we know

databases when we take databases we we

know like SQL and even know sqls so we

know databases right predominantly SQL

and then version control system so any

developers like nowadays like software

engineer or a data engineer or a data

scientist should know Version Control

System like get GitHub bit bucket and

everything so we already aware of it

right and then uh devops so devops

includes like kubernets Docker or

workflow schedulers like airflow or any

other automation schedulers so which is

we are currently working on and we are

somehow we are using it in our job so

that is also fine we are doing it and

then operating system Linux of course we

as a data Engineers we work with Linux

right predominantly Linux is everything

for us software development and

methodologies yeah we are working in

agile scrum so we are aware of it

testing and debugging so anyway we are

doing testing and debugging even we

write a spar job or a SQL job we do test

we write test Frameworks and we do

testing and debugging it's part of our

any developer life and then cloud

computing so of course nowadays people

are already started adding cloud

computing as part of their skill set and

people are working with Microsoft Azu

Google cloud and Google Cloud again data

Pro bigquery so we anyways like we are

in and out of cloud as well so what is

not there with us out of this list is

API services so we are not into API we

don't know how to write an API and what

is an API first of all so that is very

important nowadays and I'll tell you why

people are asking this so that is the uh

the core part of the video actually and

then web development so do I need to

know a web development seriously so do I

need to know HTML C CSS or JavaScript or

any web server like flask or djo so why

I need to know so I I'll give you the

answer for this and then mobile

Computing so mobile Computing is

something even a software Engineers

nowadays they are not concentrating on

that so just we can remove it from the

list okay let's come to the place like

why should I have to write an API

Services an API call and secondly why

should I have to go for web

development okay so that's why I said

you are already 80% of soft engineer

right because you are a 100 percentage

of data engineer right so now if you

take companies right they started

building products and you you can say

like yeah I know people are building

products okay this products is all about

developer perspective so here the end

users are Developer within the

organization it's not outside the world

outside the end user is not like like

non- Tech person so we are building

products for the developers who work

with us within our organization so this

is happening this always happens in

Anyan parall people will build like

continuous deployment Frameworks data

movement and replication Frameworks or

they used to create this 360° data

product and then DM and so many other

stuff people create so these are not

business benefit projects or products so

we create a product within our company

people can use it for example you have a

database name and you need to know how

many uh replication has been there for

this database or you have a table name

and you need to know this table belongs

to which database and which which how

many replications we do have for this so

build a product for this when you search

for the table name you will get all the

details about the table whether it could

be my SQL or Terra data or no SQL

database whatever it is so I created a

product so now similar to this people

are building so many products like data

movement and replication so you want a

copy of the source table from your

database so you build a UI in which I'll

get the request from the user saying

that I need this table name uh one

replic in my system so I give all this I

fill all this information and in backend

it creates a schedule for this and then

it will start creating the movement data

movement and it also create the

application so these are all the some

products companies are developing

internally for the developers to easily

complete their work so now this is also

become a very good part of the project

and people are allocating budget for

this and now when you are going for an

interview right so the entire team is

almost like 15 members in your teamr

data engineers and new people are

started building such products and for

just for uh the building a front end

with HTML CSS or basic uh web UI because

this is an internal you don't want to

perfectly design a very creative UI and

all it's not required a basic uh UA is

wide up so for this we cannot go for uh

hiring right so we cannot go for a

software engineer or a frontend

developer right or an API service

writing an API service just for this I

I'm not going to hire a software

engineer with my budget what I have so

now people are expecting uh data

Engineers should know some of the skill

set especially it comes to AP services

and basic web development information

just if you spend one or two hour of

time in Internet and YouTube you will

get all these so this is what like

people are expecting okay so how much

this is going to impact in my interview

if I say I don't know uh HTML CSS or I

don't know API service or any other

skill set that a software engineer knows

but I don't know but they asking me in

the data engineering uh interview so

this is not impacting a lot okay this is

not at all impacting they are just

considering this knowledge 10 10 to 20%

from you and if you still say no I don't

know but I can able to learn they are

ready so they are going to judge you

select you based on the data engineering

skill set and how you performed in your

data engineering skill set in your

interview but why I'm telling this to

you right so uh preparing this is

actually going to be an add-on for you

so you are not going to lose any

opportunities and you may think I'm I'm

I I'm already tired of learning all

these new new stuff but that is how

system works right so we are in open

source world and like any new text tack

comes and when we see okay that if I add

it as part of my skill set and I'm going

to get more uh visibility then obviously

we have to do it okay but our primary

stack is still data engineering right

end of the day the motto company's motto

is cost saving so if I know that skill

already then it's going to be an added

advantage and even if you see in my team

people are building gen it's an internal

product so if you know how to create a

gen very basic thing and then you can

able to apply that in your project and

if you know some visualization tools and

you know how to uh project that as part

of your data and that again going to

give you some added advantage and that's

what I wanted to tell you so the agenda

of the video is completed and thanks for

watching if you really like this video

please do subscribe my channel and

forward this to your friends and

colleagues and please do share and tag

me in LinkedIn as well and I do have an

Instagram page called the data Tech

where I used to upload lot of short res

videos which will be very useful so

please do follow me over there thanks

for watching