Being A Data Engineer: Expectations vs Reality

Seattle Data Guy
2 Apr 202107:27

Summary

TLDRIn this video, Ben Rogue John, known as the Seattle Data Guy, shares his journey and the realities of being a data engineer. He contrasts his initial expectations—using advanced tools like Hadoop and Spark—with the actual industry practice of relying on older, low-code solutions like SSIS. He discusses the frequent involvement in data migrations, the behind-the-scenes nature of the role, and the unexpectedly high salary potential. This video offers valuable insights for aspiring data engineers about the true landscape of the profession.

Takeaways

  • 😀 The speaker, Ben Rogue John, shares his initial expectations and the reality of being a data engineer.
  • 🔧 He initially thought that data engineers extensively use tools like Hadoop, Spark, and Kafka, but found that many companies still use older technologies and low-code experiences.
  • 📈 He expected to write a lot of MapReduce jobs, but discovered that many data engineering tasks involve SQL interfaces and ETL tools like SSIS.
  • 🔄 The speaker emphasizes the commonality of migrations in data engineering roles, which often involve moving data or systems from one technology to another.
  • 💡 He learned that using complex systems like distributed systems and streaming analytics is not as simple as it seems and requires significant maintenance.
  • 🏢 Companies often migrate their tech stacks to find better solutions or to save costs, which can be a significant part of a data engineer's job.
  • 🤔 Ben initially thought data engineers would get more recognition, but found that the role is often behind the scenes and less celebrated than data scientists.
  • 💼 The speaker points out that data engineering roles are more abundant than data scientist positions, indicating a higher demand for data engineers.
  • 💰 Contrary to his initial salary expectations, Ben found that there are opportunities for data engineers to earn well above his initial projections if they find the right company and role.
  • 📊 He suggests that the salary for data engineers can vary widely depending on the company and location, with some companies willing to pay a premium for skilled data engineers.
  • 📈 Ben encourages viewers to share their own expectations versus reality experiences in the comments, indicating an interest in community insights and shared experiences.

Q & A

  • What were Ben Rogue John's initial expectations when he decided to become a data engineer?

    -Ben Rogue John expected to be using advanced tools like Hadoop, Spark, and Kafka, and to be writing a lot of MapReduce jobs and coding for real-time analytics and complex data systems.

  • What did Ben find out about the use of tools in data engineering roles?

    -He discovered that many companies still use older technologies and some even rely on drag-and-drop tools for ETL and data pipeline development, rather than the complex systems he initially expected.

  • What was Ben's experience with SQL Server Integration Services (SSIS) in his first job?

    -At his first job, Ben used a lot of SSIS, which involved drag-and-drop coding rather than writing code, and was very focused on Microsoft technologies.

  • Why do companies sometimes abstract away complex data systems?

    -Companies abstract away complex data systems to make it easier to find personnel with the necessary skills, as SQL is more commonly known than direct interaction with systems like Hadoop.

  • What was one of the unexpected aspects of data engineering work that Ben learned about?

    -Ben learned that a significant part of a data engineer's work involves migrations, such as code, design, or system migrations, which can be quite common in the industry.

  • How often did Ben find himself working on migrations during his career?

    -Ben found that he worked on migrations at least once every two years, indicating that it's a regular part of the data engineering role.

  • What is the general perception of data engineers compared to data scientists?

    -Data engineers tend to get less fanfare and recognition compared to data scientists, whose work is often more tangible and noticeable to executives and management.

  • What was Ben's initial salary expectation as a data engineer five years into his career?

    -Ben initially expected to make around 100 to 120k in salary about five years into his career as a data engineer.

  • What did Ben find out about salary opportunities for data engineers?

    -Ben found that there are many opportunities where data engineers can earn far more than his initial expectation, especially if they find the right company and role.

  • What advice does Ben give to those looking to become a data engineer based on his experiences?

    -Ben advises that understanding the realities of the role, including the prevalence of migrations and the behind-the-scenes nature of the work, is important for those considering a career in data engineering.

Outlines

plate

Dieser Bereich ist nur für Premium-Benutzer verfügbar. Bitte führen Sie ein Upgrade durch, um auf diesen Abschnitt zuzugreifen.

Upgrade durchführen

Mindmap

plate

Dieser Bereich ist nur für Premium-Benutzer verfügbar. Bitte führen Sie ein Upgrade durch, um auf diesen Abschnitt zuzugreifen.

Upgrade durchführen

Keywords

plate

Dieser Bereich ist nur für Premium-Benutzer verfügbar. Bitte führen Sie ein Upgrade durch, um auf diesen Abschnitt zuzugreifen.

Upgrade durchführen

Highlights

plate

Dieser Bereich ist nur für Premium-Benutzer verfügbar. Bitte führen Sie ein Upgrade durch, um auf diesen Abschnitt zuzugreifen.

Upgrade durchführen

Transcripts

plate

Dieser Bereich ist nur für Premium-Benutzer verfügbar. Bitte führen Sie ein Upgrade durch, um auf diesen Abschnitt zuzugreifen.

Upgrade durchführen
Rate This

5.0 / 5 (0 votes)

Ähnliche Tags
Data EngineeringCareer InsightsTools OverviewETL ProcessesHadoopSparkKafkaSQL ServerMigrationsSalary RangeTech Trends
Benötigen Sie eine Zusammenfassung auf Englisch?