Why I Left Data Science - And Picked Data Engineering Instead

Summary

The speaker reflects on their journey from data science to data engineering, inspired initially by the allure of data science as the 'sexiest job' in the 21st century. Their college experience with epidemiology and computer science led to a hospital network job, where the lack of project direction soured their experience. A subsequent role in healthcare analytics exposed them to product development and the value of data engineering. The speaker found satisfaction in creating permanent solutions and products, rather than endless research cycles, and discusses the broader trend of data scientists transitioning into data engineering roles due to the demands of data infrastructure and pipeline reliability.

Takeaways

• 📈 The speaker was initially inspired to pursue data science after reading an HBR article in 2012 that dubbed it the 'sexiest job of the 21st century'.
• 🏫 Their college experience in epidemiology and computer science courses, particularly the application of data in healthcare, steered them towards data science.
• 🔍 The speaker learned about the importance of data in understanding disease spread and management, exemplified by the historical case of John Snow and the cholera outbreak.
• 💼 The first job in a hospital network as an analyst led to a push towards involvement in data science projects, specifically focusing on patient readmission issues.
• 🤔 A lack of project direction and the newness of the field among data scientists at the hospital left the speaker with a less than ideal experience.
• 🛠️ Transitioning to a healthcare analytics company, the speaker witnessed the application of data science in building solid products, such as fraud detection and opioid prescription monitoring.
• 🔄 The desire to make research permanent and avoid being stuck in endless research loops led to an interest in data engineering.
• 🔧 Data engineering was seen as a larger problem set, often requiring the creation of reliable data pipelines and infrastructure, which many data scientists end up doing.
• 💡 The realization that much of the work in data involves infrastructure and pipelines, which are essential for reliable data, contributed to the shift towards data engineering.
• 🔄 The speaker's preference for work that delivers a permanent product, such as a well-structured database or table, contrasted with the temporary nature of research projects.
• 🏭 There is a need for better-defined roles and company culture to ensure that people can focus on the work they enjoy and are good at, whether it's data science or data engineering.

Q & A

What was the impact of the HBR article on the speaker's career choice in 2012?

The HBR article titled 'Data Science: The Sexiest Job of the 21st Century' influenced the speaker to pursue a career in data science.

What branch of medicine did the speaker study in college that involved applied data?

The speaker studied epidemiology in college, which is a branch of medicine that focuses on the mathematical and statistical aspects of disease control and management.

Who is the Jon Snow mentioned in the script, and what did he discover about cholera?

The script refers to a historical figure named John Snow, who discovered that cholera was spread through contaminated water, not bad air as previously thought.

What motivated the speaker to combine computer science and statistics for healthcare applications?

The speaker's interest in using computer science and statistics for healthcare was motivated by their college courses in epidemiology and computer science.

What was the speaker's first job related to data science, and what was their initial role?

The speaker's first job related to data science was at a hospital network, where they were initially hired as an analyst but aspired to work as a data scientist.

What healthcare problem did the speaker work on at the hospital network?

The speaker worked on the problem of patient readmissions, using socioeconomic data to detect factors that might contribute to this issue.

What was the speaker's experience like working with data scientists who were new to the field?

The speaker did not have a great experience because the data scientists, including themselves, were new to the field and did not know how to effectively drive a project forward.

What made the speaker shift towards data engineering at the healthcare analytics company?

The speaker was influenced to shift towards data engineering after seeing how data science and basic stats could be used to drive value and build solid products in the healthcare analytics company.

What are some of the common issues that the speaker observed in data science projects?

The speaker observed that many companies did not know how to use data or find value from it, and that data scientists often had to perform data engineering tasks to ensure clean and reliable datasets.

Why did the speaker prefer data engineering over data science?

The speaker preferred data engineering because it involved creating more permanent solutions and products, and they enjoyed the process of making models more performant and delivering tangible outcomes.

What advice does the speaker have for people considering a career in data science or data engineering?

The speaker suggests that people should consider their preferences for the type of work they enjoy, whether it's research-focused or more about creating permanent solutions, and also consider the necessity and demands of the data world.