Agile in Data Science: A case study

Summary

Snikda Sathi, a Senior Business Analyst at News UK, discusses her experience implementing Agile within the data science team. She highlights challenges such as poor stakeholder alignment, unclear communication, and difficulty estimating project timelines. By adapting Agile principles like sprints, standups, and retrospectives, the team improved focus, transparency, and stakeholder relationships. The shift to Agile fostered better collaboration and led to improved motivation and productivity. Despite the inherent complexity of data science, Agile’s flexible approach allowed the team to deliver meaningful results and win industry recognition.

Takeaways

• 😀 **Agile Methodology Applied to Data Science:** Implementing agile in data science teams is a challenge due to the iterative and experimental nature of the work. It requires adapting agile practices like sprints and stand-ups to suit the data science environment.
• 😀 **Communication as Key:** Daily stand-ups and sprint planning meetings improved communication between team members and helped clarify priorities, resulting in a more focused and organized workflow.
• 😀 **Business Involvement in Backlog Prioritization:** A dedicated business analyst worked with the senior data scientist to prioritize the backlog, ensuring that tasks were aligned with the most important business objectives.
• 😀 **Sprints and Demos in Data Science:** Data science teams face challenges with demos due to the abstract nature of their work (e.g., training models), but sharing progress through sprint demos was key for visibility and stakeholder buy-in.
• 😀 **Frequent Retrospectives:** Regular retrospectives were held to address pain points such as work overload and off-hours requests from stakeholders, enabling continuous improvement in the team’s processes.
• 😀 **Challenges in Estimation:** Estimating story points or timelines in data science is difficult due to the experimental nature of the work. Agile methods had to be adjusted for this challenge, focusing on progress rather than precise estimates.
• 😀 **Stakeholder Engagement:** Stakeholder requests often changed midway through sprints, which posed challenges. However, the team learned to adapt to these changes without derailing their progress.
• 😀 **Morale Boost Through Agile:** Adopting agile practices helped improve the morale of the data science team, as it made goals clearer and more achievable, leading to greater satisfaction and reduced frustration.
• 😀 **Iterative Model Development:** Data scientists often feel strongly about their models, which leads to in-depth investigations. Agile methods helped bring a practical focus on feasibility, helping to stop unproductive work early.
• 😀 **Cross-Location Teamwork:** Teams spread across multiple locations (e.g., London and Bangalore) faced unique communication challenges, but agile practices like stand-ups and regular retrospectives helped manage these dynamics effectively.

Q & A

What were the initial challenges faced by the data science team at News UK before implementing agile?

The data science team faced several challenges, including a lack of focus, unclear priorities, and poor communication. Stakeholders were often giving direct tasks to the data scientists, leading to chaotic workflows, missed deadlines, and low morale.

How did the speaker address the challenges within the data science team?

The speaker introduced agile methodologies, including backlog prioritization, daily standups, sprint planning, and retrospectives. These practices helped clarify goals, align team efforts, and improve communication and focus within the team.

Why was it difficult to estimate tasks using story points in data science?

Estimating tasks using story points was difficult in data science because the work is experimental and iterative by nature. Data scientists often work on models that are unpredictable, and the outcome of their efforts cannot be precisely determined at the start of the sprint.

What was the main goal of the agile transformation for the data science team?

The main goal of the agile transformation was to improve communication, focus, and productivity within the data science team. By using agile practices, the speaker aimed to align the team’s work with organizational priorities and manage stakeholder expectations.

How did the implementation of agile impact the team's morale and productivity?

Implementing agile practices helped boost the team's morale by providing clearer structure, setting realistic goals, and enabling the team to demonstrate tangible progress. As a result, the team became more productive and gained stakeholder confidence.

What key agile practices did the speaker use to improve the team’s workflow?

The speaker used several key agile practices, such as backlog prioritization, daily standups, sprint planning, and retrospectives. These practices allowed the team to assess progress regularly, adjust their priorities, and collaborate more effectively.

How did the data science team handle deliverables in agile sprints?

Since data science work often involves iterative experimentation, the team focused on delivering parts of a model or data transformations that could be tested within short sprints. This helped demonstrate progress and feasibility without the pressure to complete an entire model in one sprint.

What was the importance of retrospectives in the agile process for the data science team?

Retrospectives were crucial for improving team processes and communication. They allowed the team to reflect on what worked and what didn’t, identify bottlenecks, and adjust their approach to ensure smoother workflows in future sprints.

What role did empathy play in the agile transformation of the team?

Empathy played a key role in the agile transformation. The speaker emphasized one-on-one conversations and understanding the team’s challenges and concerns, which helped foster trust and encourage collaboration during the agile adoption process.

What advice did the speaker offer regarding applying agile in a data science context?

The speaker advised adapting agile practices to fit the unique needs of data science teams. This includes focusing on delivering parts of models or data processes that can be demonstrated in sprints, and not forcing standard agile practices like story point estimation when they don't make sense for the work.