Data Engineers = 80% Software Engineers

Summary

This video explores the evolving skill set expectations in the data engineering job market from 2024 onwards. It highlights the increasing demand for software engineering skills like API services and basic web development, in addition to traditional data engineering competencies. The speaker emphasizes the importance of adapting to these changes to enhance job prospects and stay relevant in the industry.

Takeaways

• 😀 The video discusses the evolving skill set expectations in the data engineering job market, particularly from 2024 onwards.
• 🔍 Predictions from 2020-2023 suggested that AI and data science skills would be crucial for data engineers, but current trends indicate a shift.
• 💼 The job market now expects data engineers to have a broader set of skills, including software engineering competencies.
• 👨‍💻 Primary skills of software engineers include programming languages, SQL, data structures and algorithms, databases, version control, software development life cycle, web development, cloud computing, testing, debugging, and mobile development.
• 🤔 Data engineers typically already possess many of these skills, such as programming languages, data structures and algorithms, databases, version control, DevOps, operating systems (Linux), and cloud computing.
• 🚀 However, data engineers may lack skills in API services and basic web development, which are becoming increasingly important.
• 🏢 Companies are building internal products for developers, which often require basic web UIs and API services, leading to a demand for these skills in data engineers.
• 💡 The video suggests that while these additional skills are not a major requirement, having them can be an advantage and can increase job opportunities.
• 📈 The impact of not knowing these additional skills in interviews is minimal, as the primary focus remains on core data engineering skills.
• 📚 Learning new skills like basic web development and API services can be done with minimal effort and can provide an edge in the job market.
• 🌐 The speaker encourages viewers to stay updated with new skills and technologies, as the tech industry is constantly evolving.

Q & A

What is the main focus of the video?

The video focuses on the evolving skill set requirements for data engineers, particularly highlighting the increasing expectation of software engineering skills in data engineering roles.

What was the prediction for the data engineering job market in 2020-2025?

The prediction suggested that people should learn AI and data science and add these to their data engineering skill set to increase job opportunities.

What has changed in the data engineering job market from 2024 onwards?

From 2024, the data engineering job market has started to expect more software engineering skills as part of the job description (JD).

What are some primary skills of software engineers mentioned in the video?

The primary skills mentioned include programming languages, SQL, data structures and algorithms, databases, version control systems, software development life cycle, methodologies like Agile and Scrum, web development, cloud computing, testing and debugging, and operating systems like Linux.

Which skills do data engineers typically already possess according to the video?

Data engineers typically already possess skills in programming languages, data structures and algorithms, databases (especially SQL), version control systems, DevOps (including Kubernetes and Docker), operating systems (like Linux), software development methodologies (like Agile and Scrum), and cloud computing.

What skills are data engineers currently lacking according to the video?

Data engineers are currently lacking skills in API services and web development, as these are not typically part of their skill set.

Why are companies expecting data engineers to know API services and basic web development?

Companies are building internal products for developers, which often require a basic UI for user interaction. Data engineers are expected to have some knowledge in these areas to contribute to these projects without the need for hiring additional software engineers or frontend developers.

How does the video suggest data engineers can acquire knowledge in API services and web development?

The video suggests that spending a few hours on the internet and YouTube can help data engineers learn the basics of API services and web development.

What impact does not knowing HTML, CSS, or API services have on a data engineer's interview?

Not knowing these skills does not significantly impact the interview outcome. Interviewers are primarily interested in the candidate's data engineering skills, but having some knowledge in these areas can be an added advantage.

What is the advice given to data engineers regarding learning new skills in the video?

The advice is to be open to learning new skills, such as API services and basic web development, as this can increase visibility and opportunities in the job market. It is also beneficial for cost-saving within companies.