Being A Data Engineer: Expectations vs Reality

Summary

In this video, Ben Rogue John, known as the Seattle Data Guy, shares his journey and the realities of being a data engineer. He contrasts his initial expectations—using advanced tools like Hadoop and Spark—with the actual industry practice of relying on older, low-code solutions like SSIS. He discusses the frequent involvement in data migrations, the behind-the-scenes nature of the role, and the unexpectedly high salary potential. This video offers valuable insights for aspiring data engineers about the true landscape of the profession.

Takeaways

• 😀 The speaker, Ben Rogue John, shares his initial expectations and the reality of being a data engineer.
• 🔧 He initially thought that data engineers extensively use tools like Hadoop, Spark, and Kafka, but found that many companies still use older technologies and low-code experiences.
• 📈 He expected to write a lot of MapReduce jobs, but discovered that many data engineering tasks involve SQL interfaces and ETL tools like SSIS.
• 🔄 The speaker emphasizes the commonality of migrations in data engineering roles, which often involve moving data or systems from one technology to another.
• 💡 He learned that using complex systems like distributed systems and streaming analytics is not as simple as it seems and requires significant maintenance.
• 🏢 Companies often migrate their tech stacks to find better solutions or to save costs, which can be a significant part of a data engineer's job.
• 🤔 Ben initially thought data engineers would get more recognition, but found that the role is often behind the scenes and less celebrated than data scientists.
• 💼 The speaker points out that data engineering roles are more abundant than data scientist positions, indicating a higher demand for data engineers.
• 💰 Contrary to his initial salary expectations, Ben found that there are opportunities for data engineers to earn well above his initial projections if they find the right company and role.
• 📊 He suggests that the salary for data engineers can vary widely depending on the company and location, with some companies willing to pay a premium for skilled data engineers.
• 📈 Ben encourages viewers to share their own expectations versus reality experiences in the comments, indicating an interest in community insights and shared experiences.

Q & A

What were Ben Rogue John's initial expectations when he decided to become a data engineer?

Ben Rogue John expected to be using advanced tools like Hadoop, Spark, and Kafka, and to be writing a lot of MapReduce jobs and coding for real-time analytics and complex data systems.

What did Ben find out about the use of tools in data engineering roles?

He discovered that many companies still use older technologies and some even rely on drag-and-drop tools for ETL and data pipeline development, rather than the complex systems he initially expected.

What was Ben's experience with SQL Server Integration Services (SSIS) in his first job?

At his first job, Ben used a lot of SSIS, which involved drag-and-drop coding rather than writing code, and was very focused on Microsoft technologies.

Why do companies sometimes abstract away complex data systems?

Companies abstract away complex data systems to make it easier to find personnel with the necessary skills, as SQL is more commonly known than direct interaction with systems like Hadoop.

What was one of the unexpected aspects of data engineering work that Ben learned about?

Ben learned that a significant part of a data engineer's work involves migrations, such as code, design, or system migrations, which can be quite common in the industry.

How often did Ben find himself working on migrations during his career?

Ben found that he worked on migrations at least once every two years, indicating that it's a regular part of the data engineering role.

What is the general perception of data engineers compared to data scientists?

Data engineers tend to get less fanfare and recognition compared to data scientists, whose work is often more tangible and noticeable to executives and management.

What was Ben's initial salary expectation as a data engineer five years into his career?

Ben initially expected to make around 100 to 120k in salary about five years into his career as a data engineer.

What did Ben find out about salary opportunities for data engineers?

Ben found that there are many opportunities where data engineers can earn far more than his initial expectation, especially if they find the right company and role.

What advice does Ben give to those looking to become a data engineer based on his experiences?

Ben advises that understanding the realities of the role, including the prevalence of migrations and the behind-the-scenes nature of the work, is important for those considering a career in data engineering.