Data Engineers = 80% Software Engineers
Summary
TLDRThis video explores the evolving skill set expectations in the data engineering job market from 2024 onwards. It highlights the increasing demand for software engineering skills like API services and basic web development, in addition to traditional data engineering competencies. The speaker emphasizes the importance of adapting to these changes to enhance job prospects and stay relevant in the industry.
Takeaways
- 😀 The video discusses the evolving skill set expectations in the data engineering job market, particularly from 2024 onwards.
- 🔍 Predictions from 2020-2023 suggested that AI and data science skills would be crucial for data engineers, but current trends indicate a shift.
- 💼 The job market now expects data engineers to have a broader set of skills, including software engineering competencies.
- 👨💻 Primary skills of software engineers include programming languages, SQL, data structures and algorithms, databases, version control, software development life cycle, web development, cloud computing, testing, debugging, and mobile development.
- 🤔 Data engineers typically already possess many of these skills, such as programming languages, data structures and algorithms, databases, version control, DevOps, operating systems (Linux), and cloud computing.
- 🚀 However, data engineers may lack skills in API services and basic web development, which are becoming increasingly important.
- 🏢 Companies are building internal products for developers, which often require basic web UIs and API services, leading to a demand for these skills in data engineers.
- 💡 The video suggests that while these additional skills are not a major requirement, having them can be an advantage and can increase job opportunities.
- 📈 The impact of not knowing these additional skills in interviews is minimal, as the primary focus remains on core data engineering skills.
- 📚 Learning new skills like basic web development and API services can be done with minimal effort and can provide an edge in the job market.
- 🌐 The speaker encourages viewers to stay updated with new skills and technologies, as the tech industry is constantly evolving.
Q & A
What is the main focus of the video?
-The video focuses on the evolving skill set requirements for data engineers, particularly highlighting the increasing expectation of software engineering skills in data engineering roles.
What was the prediction for the data engineering job market in 2020-2025?
-The prediction suggested that people should learn AI and data science and add these to their data engineering skill set to increase job opportunities.
What has changed in the data engineering job market from 2024 onwards?
-From 2024, the data engineering job market has started to expect more software engineering skills as part of the job description (JD).
What are some primary skills of software engineers mentioned in the video?
-The primary skills mentioned include programming languages, SQL, data structures and algorithms, databases, version control systems, software development life cycle, methodologies like Agile and Scrum, web development, cloud computing, testing and debugging, and operating systems like Linux.
Which skills do data engineers typically already possess according to the video?
-Data engineers typically already possess skills in programming languages, data structures and algorithms, databases (especially SQL), version control systems, DevOps (including Kubernetes and Docker), operating systems (like Linux), software development methodologies (like Agile and Scrum), and cloud computing.
What skills are data engineers currently lacking according to the video?
-Data engineers are currently lacking skills in API services and web development, as these are not typically part of their skill set.
Why are companies expecting data engineers to know API services and basic web development?
-Companies are building internal products for developers, which often require a basic UI for user interaction. Data engineers are expected to have some knowledge in these areas to contribute to these projects without the need for hiring additional software engineers or frontend developers.
How does the video suggest data engineers can acquire knowledge in API services and web development?
-The video suggests that spending a few hours on the internet and YouTube can help data engineers learn the basics of API services and web development.
What impact does not knowing HTML, CSS, or API services have on a data engineer's interview?
-Not knowing these skills does not significantly impact the interview outcome. Interviewers are primarily interested in the candidate's data engineering skills, but having some knowledge in these areas can be an added advantage.
What is the advice given to data engineers regarding learning new skills in the video?
-The advice is to be open to learning new skills, such as API services and basic web development, as this can increase visibility and opportunities in the job market. It is also beneficial for cost-saving within companies.
Outlines
💼 Evolving Skill Sets in Data Engineering
This paragraph discusses the changing landscape of data engineering job market and the evolving skill set expectations. Initially, it highlights the importance of AI and data science in the past, but shifts focus to the current trend where software engineering skills are increasingly being integrated into data engineering roles. The speaker emphasizes the need for data engineers to be proficient in programming languages, SQL, data structures, algorithms, databases, version control systems, software development life cycles, methodologies like Agile and Scrum, web development, cloud computing, testing, debugging, and operating systems like Linux. It also points out that while data engineers are expected to have some knowledge of API services and web development, these are not core requirements but rather additional skills that can enhance their profile. The speaker concludes by stating that the primary focus should remain on data engineering, but having a broader skill set can be advantageous.
🏢 Internal Product Development and Its Impact on Job Market
The second paragraph delves into how companies are building internal products for developers, which are essentially tools and frameworks that facilitate data movement, replication, and management. These products are designed to be user-friendly for developers within the organization, reducing the need for external hiring for specific tasks like front-end development or API services. The speaker explains that while these skills are not mandatory, having a basic understanding of HTML, CSS, and web development can be beneficial. The paragraph also touches on the interview process, where these additional skills are considered but not critical. The speaker suggests that even if a candidate lacks these skills, their ability to learn and adapt is more important. The emphasis is on the cost-effectiveness of having multi-skilled employees and the potential for data engineers to contribute to internal product development, thereby increasing their value within the company.
Mindmap
Keywords
💡Data Engineering
💡Software Engineering
💡AI and Data Science
💡Job Market Predictions
💡Programming Languages
💡SQL
💡Version Control Systems
💡DevOps
💡API Services
💡Web Development
💡Cloud Computing
Highlights
Data engineers and software engineers are increasingly expected to have overlapping skill sets.
In 2022 and 2023, the job market predicted a need for AI and data science skills in data engineering roles.
From 2024 onwards, the data engineering job market is expecting additional skills beyond AI and data science.
Software engineering skills are now part of the expectations in data engineering job descriptions.
Primary skills of software engineers include programming languages, SQL, data structures, algorithms, databases, version control, and software development methodologies.
Data engineers typically already possess skills in programming languages, data structures, algorithms, databases, and version control systems.
DevOps skills, including Kubernetes, Docker, and workflow schedulers like Airflow, are also commonly known among data engineers.
Linux operating system proficiency is a must-have for data engineers.
Cloud computing skills are increasingly being integrated into data engineers' skill sets.
API services and web development are areas where data engineers might lack expertise and are now being sought after in the job market.
Understanding and creating APIs is becoming crucial for data engineers as companies build internal developer-focused products.
Basic web development skills, such as HTML, CSS, and JavaScript, are being expected for building internal tools and UIs.
Mobile development is not a focus area even for software engineers currently, and can be excluded from the essential skills list for data engineers.
Companies are building products for internal use by developers, such as data movement and replication frameworks.
Data engineers are expected to have some knowledge of front-end development to contribute to these internal product developments.
Interviewers are considering knowledge of software engineering skills as 10-20% of the evaluation criteria for data engineering roles.
Even without prior knowledge of certain software engineering skills, data engineers can still be considered if they show a willingness to learn.
Learning additional skills like API services and basic web development can be an add-on and provide an advantage in the job market.
Data engineers are encouraged to stay updated with new technologies and integrate them into their skill sets for better job opportunities.
The speaker has an Instagram page called 'the data Tech' where they upload useful short videos related to data technology.
Transcripts
hi welcome to the data Tech 100% of data
Engineers or 80% of software Engineers
100% of software Engineers are just 50%
of data engineer okay what is this all
about you're saying something like a
calculation okay so this video is going
to be completely an interesting fact so
please do watch this
completely so roughly in 2022 and
2023 the prediction of data engineering
job market and work Market in 2020 25
and 2026 was like okay people should
learn uh AI data science and they have
to add this as part of their data
engineering skill set so then only like
we will be getting lot of job
opportunities and data engineering but
what happening now because always the
prediction may it change so what is
happening now from the beginning of 2024
right the data engineering job market
have adding few more skill set as part
of the expectation in the JD so what is
that is that AI data science no it's not
like it's not that
actually so recently so I do have many
contacts in my LinkedIn and people used
to share their interview experience with
me and many of my friends and my ex-
colleagues have attended lot of
interviews in the recent times so this
is not just within this month it's it
happened I I've been I have been doing
this research for almost last three
months with my connections and what I
come to know is people are expecting
software engineering skill set as part
of the data engineering skill skill set
okay so now as a data engineer how
should I know whether what are all the
skill sets or software engineering and
whether do I already know that or not so
that's why I started with 100% data
Engineers or 80% software Engineers
already okay so what are the primary
skill set of software Engineers okay
let's start doing the list programming
language and SQL and then you need to
know data structures andal algorithms
and then databases Version Control
System Dev apps software development
life cycle and methodology like you need
to know the zajil methodology scrum and
all those stuff web development cloud
computing testing and debugging so you
need to know API services and how to
create an API how to write an API in any
any coding it could be on what is API in
first place and then you need to know
operating system especially Linux and
last mobile development so these are all
the some of the primary skill set for
software engineer a typical software an
average software engineer should know
all this as part of their skill okay so
now as a data engineer what I know
already from this list is very maximum
right if you see we know programming
languages and we know data structures
and algorithms right and we know
databases when we take databases we we
know like SQL and even know sqls so we
know databases right predominantly SQL
and then version control system so any
developers like nowadays like software
engineer or a data engineer or a data
scientist should know Version Control
System like get GitHub bit bucket and
everything so we already aware of it
right and then uh devops so devops
includes like kubernets Docker or
workflow schedulers like airflow or any
other automation schedulers so which is
we are currently working on and we are
somehow we are using it in our job so
that is also fine we are doing it and
then operating system Linux of course we
as a data Engineers we work with Linux
right predominantly Linux is everything
for us software development and
methodologies yeah we are working in
agile scrum so we are aware of it
testing and debugging so anyway we are
doing testing and debugging even we
write a spar job or a SQL job we do test
we write test Frameworks and we do
testing and debugging it's part of our
any developer life and then cloud
computing so of course nowadays people
are already started adding cloud
computing as part of their skill set and
people are working with Microsoft Azu
Google cloud and Google Cloud again data
Pro bigquery so we anyways like we are
in and out of cloud as well so what is
not there with us out of this list is
API services so we are not into API we
don't know how to write an API and what
is an API first of all so that is very
important nowadays and I'll tell you why
people are asking this so that is the uh
the core part of the video actually and
then web development so do I need to
know a web development seriously so do I
need to know HTML C CSS or JavaScript or
any web server like flask or djo so why
I need to know so I I'll give you the
answer for this and then mobile
Computing so mobile Computing is
something even a software Engineers
nowadays they are not concentrating on
that so just we can remove it from the
list okay let's come to the place like
why should I have to write an API
Services an API call and secondly why
should I have to go for web
development okay so that's why I said
you are already 80% of soft engineer
right because you are a 100 percentage
of data engineer right so now if you
take companies right they started
building products and you you can say
like yeah I know people are building
products okay this products is all about
developer perspective so here the end
users are Developer within the
organization it's not outside the world
outside the end user is not like like
non- Tech person so we are building
products for the developers who work
with us within our organization so this
is happening this always happens in
Anyan parall people will build like
continuous deployment Frameworks data
movement and replication Frameworks or
they used to create this 360° data
product and then DM and so many other
stuff people create so these are not
business benefit projects or products so
we create a product within our company
people can use it for example you have a
database name and you need to know how
many uh replication has been there for
this database or you have a table name
and you need to know this table belongs
to which database and which which how
many replications we do have for this so
build a product for this when you search
for the table name you will get all the
details about the table whether it could
be my SQL or Terra data or no SQL
database whatever it is so I created a
product so now similar to this people
are building so many products like data
movement and replication so you want a
copy of the source table from your
database so you build a UI in which I'll
get the request from the user saying
that I need this table name uh one
replic in my system so I give all this I
fill all this information and in backend
it creates a schedule for this and then
it will start creating the movement data
movement and it also create the
application so these are all the some
products companies are developing
internally for the developers to easily
complete their work so now this is also
become a very good part of the project
and people are allocating budget for
this and now when you are going for an
interview right so the entire team is
almost like 15 members in your teamr
data engineers and new people are
started building such products and for
just for uh the building a front end
with HTML CSS or basic uh web UI because
this is an internal you don't want to
perfectly design a very creative UI and
all it's not required a basic uh UA is
wide up so for this we cannot go for uh
hiring right so we cannot go for a
software engineer or a frontend
developer right or an API service
writing an API service just for this I
I'm not going to hire a software
engineer with my budget what I have so
now people are expecting uh data
Engineers should know some of the skill
set especially it comes to AP services
and basic web development information
just if you spend one or two hour of
time in Internet and YouTube you will
get all these so this is what like
people are expecting okay so how much
this is going to impact in my interview
if I say I don't know uh HTML CSS or I
don't know API service or any other
skill set that a software engineer knows
but I don't know but they asking me in
the data engineering uh interview so
this is not impacting a lot okay this is
not at all impacting they are just
considering this knowledge 10 10 to 20%
from you and if you still say no I don't
know but I can able to learn they are
ready so they are going to judge you
select you based on the data engineering
skill set and how you performed in your
data engineering skill set in your
interview but why I'm telling this to
you right so uh preparing this is
actually going to be an add-on for you
so you are not going to lose any
opportunities and you may think I'm I'm
I I'm already tired of learning all
these new new stuff but that is how
system works right so we are in open
source world and like any new text tack
comes and when we see okay that if I add
it as part of my skill set and I'm going
to get more uh visibility then obviously
we have to do it okay but our primary
stack is still data engineering right
end of the day the motto company's motto
is cost saving so if I know that skill
already then it's going to be an added
advantage and even if you see in my team
people are building gen it's an internal
product so if you know how to create a
gen very basic thing and then you can
able to apply that in your project and
if you know some visualization tools and
you know how to uh project that as part
of your data and that again going to
give you some added advantage and that's
what I wanted to tell you so the agenda
of the video is completed and thanks for
watching if you really like this video
please do subscribe my channel and
forward this to your friends and
colleagues and please do share and tag
me in LinkedIn as well and I do have an
Instagram page called the data Tech
where I used to upload lot of short res
videos which will be very useful so
please do follow me over there thanks
for watching
Weitere ähnliche Videos ansehen
"AI பழசாகிடுச்சு, Engineering படிப்பும் - 6 கேள்வியும்..?" Jayaprakash Gandhi Shocking பேட்டி
Top 17 BEST Data Science & Analytics Certificates (2024)
TEKNIK INFORMATIKA
Engineering Degree Tier List 2024 (The BEST Engineering Degrees RANKED)
Honest Reality of IT Job market in 2024 !! Is it even worth it to study so hard ?
I analyzed 2,765,739 jobs to solve THIS
5.0 / 5 (0 votes)