The Harsh Reality of Being a Data Engineer

Summary

In this video, Ben Rogue John, the Seattle data guy, addresses the harsh realities of being a data engineer. He discusses the lack of attention from software engineers to data pipelines, the trend of companies trying to eliminate data engineering roles, and the prevalence of 'data swamps' instead of well-structured data lakes. He also touches on the unrealistic expectations placed on data professionals to be experts in all areas of data work and the importance of acknowledging one's limits. Lastly, he advises not to worry about always using the latest technologies, emphasizing the value of understanding foundational concepts.

Takeaways

• 🔧 Data engineering involves dealing with the harsh realities of data pipeline maintenance and not always working on Big Data systems.
• 🛠️ Many software engineers may not be aware of how their changes can impact data pipelines, leading to the need for data contracts to ensure data integrity.
• 🔍 Companies sometimes attempt to eliminate data engineering roles, but often realize the importance of having someone manage and own data pipelines for analysts and scientists.
• 🏗️ The industry struggles with 'data swamps' where data is dumped without structure, leading to chaos and difficulty in managing and accessing it.
• 🤔 There's a misconception in companies expecting data professionals to be experts in all aspects of data work, similar to expecting a programmer to know all technology areas.
• 📈 Data engineers must understand the depth of their skills and know their limits, seeking additional training or senior assistance when necessary.
• 📚 Learning from older technologies can provide valuable insights into best practices and the evolution of data warehousing and modeling techniques.
• 🚀 Not using the latest technologies doesn't mean falling behind; it can offer a deeper understanding of where current practices and solutions originated.
• 💡 The speaker emphasizes the importance of continuous learning and sharing knowledge within the data community to improve overall understanding.
• 🌐 Data engineers should focus on mastering the basics and not worry too much about the hype around new technologies, as fundamentals are crucial for long-term success.

Q & A

What is the main topic of Ben Rogue John's video?

The main topic of Ben Rogue John's video is the harsh realities of being a data engineer.

Why might software engineers not always care about data?

Software engineers might not always care about data because their reviews and performance metrics are often focused on delivering new features and functionality, which may not take into account how these changes could impact data pipelines.

What is the purpose of data contracts in the context discussed in the video?

Data contracts are becoming important to ensure that changes made by data producers, such as software engineers, do not break data pipelines, as these changes can have a significant impact on data engineers' work.

Why did Ben Rogue John mention the development of a system at Facebook?

Ben Rogue John mentioned the development of a system at Facebook to automatically scan and detect changes in data tables from sources, ensuring that data engineers are aware of any modifications that could affect their work.

What is one reason companies might want to remove data engineering roles?

Some companies want to remove data engineering roles because they see them as a bottleneck and would prefer data analysts and scientists to have direct access to data without the need for data engineering processes.

What is the term used to describe poorly structured data storage that Ben Rogue John discussed in the video?

The term used to describe poorly structured data storage is 'data swamps,' which refers to chaotic and unorganized data storage situations.

Why do companies sometimes struggle with defining the roles of data engineers, data scientists, and data architects?

Companies sometimes struggle with defining these roles because they expect individuals in these positions to have a broad range of skills and be able to handle all data-related tasks, which is unrealistic given the complexity and specialization required in the data field.

What is the 'Iceberg' meme mentioned by Ben Rogue John regarding SQL, and what does it signify?

The 'Iceberg' meme signifies that SQL is a deep and complex skill, with many layers and nuances to understand beyond just the basic commands, and that even experienced professionals continue to learn and discover new aspects of it.

Why is it important for data professionals to know their limits and seek help when needed?

It is important for data professionals to know their limits because the data field is vast and constantly evolving, making it impossible for one person to be an expert in every area. Seeking help ensures that projects are completed efficiently and accurately.

What is the advice given by Ben Rogue John regarding the use of older technologies in data engineering?

Ben Rogue John advises that working with older technologies is not a disadvantage, as it allows professionals to understand the history and evolution of best practices in data warehousing and modeling, and to appreciate the reasons behind current approaches.

What is the final reality that Ben Rogue John discusses in the video about data engineering?

The final reality discussed is that data engineers won't always get to use the newest and most hyped technologies, but focusing on the fundamentals and understanding the evolution of the field is more valuable than chasing the latest trends.