Why i quit data engineering(I will never go back)

Summary

A former data engineer shares their personal experience transitioning from software engineering to data engineering, revealing unexpected challenges. They discuss frustrations with constant troubleshooting, the intangibility of their work, and the lack of recognition for their efforts. The speaker highlights the isolation in cross-team collaboration and the repetitive nature of the job. They also emphasize the highly technical and detail-oriented aspects of data engineering, which may not appeal to everyone. Ultimately, the speaker reflects on why they returned to software engineering, providing valuable insights for those considering a career in data engineering.

Takeaways

• 😀 Firefighting takes up a significant portion of a data engineer's time, with issues like failed Spark jobs and broken pipelines often consuming 50-60% of their workday.
• 😀 Unlike software engineering, where tangible results are visible, data engineering involves working with invisible data, making the work feel detached and less rewarding.
• 😀 Data engineering roles vary greatly across companies. Some focus solely on pipeline engineering, while others offer a broader range of tasks, such as data warehousing and business intelligence.
• 😀 Data engineers often work without much recognition, as their efforts to sanitize and prepare data are overlooked compared to the praise received by data scientists and business analysts.
• 😀 Lack of cross-team collaboration is a common issue in data engineering, as engineers typically work in isolation with little interaction with data scientists or business analysts.
• 😀 Data engineering involves a lot of detailed work with data formats, optimization techniques, and performance tweaks, making it highly technical and 'nerdy'.
• 😀 Data engineering can feel repetitive, especially when working on similar tasks like pipeline maintenance and cleaning data, leading to a sense of stagnation.
• 😀 Working with invisible data that cannot be directly seen or interacted with can lead to frustration and a sense of disconnectedness from the work.
• 😀 The data engineer role can sometimes be isolating, as the work often lacks collaboration with other teams, leading to less satisfaction and feeling like a 'cog in the machine'.
• 😀 The data engineering field is often glorified as a high-paying, exciting career, but it may not meet those expectations, with many aspects of the role being tedious and repetitive.
• 😀 A love for small, technical details is essential to enjoy data engineering. Without this passion, the role may not be fulfilling for everyone.

Q & A

What initially drew the speaker to data engineering?

The speaker was drawn to data engineering by the opportunity to work with large datasets, the potential for high earnings, and the exciting challenges associated with the field.

How long did the speaker last in data engineering before realizing it wasn't what they expected?

The speaker lasted about eight months in data engineering before realizing that the job was entirely different from what they had initially imagined.

What is the main frustration the speaker had with firefighting in data engineering?

The speaker found firefighting frustrating because it consumed 50-60% of their time dealing with system failures, like Spark jobs breaking in the middle of the night, rather than focusing on creating new things or advancing projects.

How does data engineering differ from software engineering in terms of tangible results?

In software engineering, the results are tangible, as the work involves building visible products like websites or applications. In data engineering, the work involves manipulating invisible data, which feels more detached and harder to assess in terms of contribution.

What did the speaker mean by the term 'glorified pipeline engineer'?

The speaker described their role as a 'glorified pipeline engineer' because their primary task was to build data pipelines into a data warehouse with little opportunity for diverse or challenging tasks.

What kind of recognition does the speaker believe data engineers typically receive?

The speaker believes data engineers rarely receive recognition for their work. Instead, roles like business analysts and data scientists are praised for their insights, while data engineers who prepare and clean the data are often overlooked.

How did the lack of collaboration impact the speaker's experience in data engineering?

The lack of collaboration in data engineering felt isolating. Unlike software engineering, where there is constant interaction between different teams (e.g., front-end and back-end developers), data engineers often work in silos, without much cross-team interaction.

What are the challenges associated with working in data engineering as mentioned by the speaker?

Challenges in data engineering include firefighting (resolving system failures), dealing with invisible data that can't be directly interacted with, repetitive tasks, and working in a siloed environment with limited recognition.

Why does the speaker refer to data engineering as a 'nerdy' field?

The speaker refers to data engineering as 'nerdy' because it involves highly technical details, such as optimizing file formats and reducing processing time by milliseconds, which requires a deep love for technical problem-solving.

What ultimately led the speaker to return to software engineering after their experience in data engineering?

The speaker returned to software engineering because they found it more fulfilling, with clearer tangible results and better collaboration. They felt the repetitive and isolating nature of data engineering wasn't as rewarding.