The BEST Way To Measure Software Developer Performance With Dr. Nicole Forsgren

Summary

In this insightful conversation, the speakers discuss the intersection of science and software engineering, focusing on the importance of measuring software practices. They emphasize that engineering should be grounded in scientific principles and that measurement is key to improving performance. The discussion touches on the challenges of low-latency systems and the need for accurate metrics. They also analyze the findings from the Dora report, questioning some conclusions about continuous integration and its effects on organizational outcomes. Overall, the conversation highlights the significance of actionable metrics in improving software quality and delivery speed.

Takeaways

• 😀 Software engineering should be seen as the practical application of science, rooted in scientifically rational principles.
• 😀 The distinction between 'craft' and 'engineering' is largely based on measurement, with the emphasis on continuous improvement and tracking of outcomes.
• 😀 Measurement is essential in engineering, even for seemingly simple systems, where low-latency performance must be measured with high precision (e.g., millisecond accuracy).
• 😀 All outputs of a system, such as performance data, are proxies and can be influenced by various external factors like temperature and electrical signals.
• 😀 The DORA (DevOps Research and Assessment) study is considered a scientifically valid analysis of software development practices, offering insights into the software development lifecycle.
• 😀 The DORA study is significant because it allows the software engineering community to reason about their practices using data and evidence, providing a more intellectually grounded perspective on the discipline.
• 😀 However, the DORA study's findings are not without contention, especially regarding certain correlations, such as the impact of continuous integration (CI) on organizational burnout.
• 😀 The correlation between continuous integration and negative outcomes may be a spurious correlation, with no clear theoretical basis for a direct relationship between CI and organizational performance.
• 😀 The key to the DORA study's success is its focus on actionable metrics that are both generic and meaningful to everyone involved in software development, helping teams drive improvements.
• 😀 The DORA metrics provide teams with clear tools to enhance their performance and work towards building better software faster, serving as a useful guide for continuous improvement.

Q & A

What is the speaker's perspective on the term 'software engineering'?

The speaker advocates for reclaiming the term 'software engineering' to truly represent the practical application of science, emphasizing the importance of scientific principles in engineering, rather than just coding or testing.

How does the speaker differentiate between 'craft' and 'engineering' in software development?

The speaker differentiates them based on measurement, suggesting that engineering is distinguished by the ability to measure various aspects, whereas craft may focus more on subjective skills without systematic measurement.

What is the significance of measurement in software engineering, according to the speaker?

Measurement is crucial for assessing the effectiveness of software development practices. The speaker highlights that while some things are harder to measure, it is still essential to attempt measurement to improve processes and ensure quality.

Why is measuring performance in low-latency systems challenging?

In low-latency systems, the challenge lies in accurately measuring response times, especially when outliers can drastically affect the results, making even millisecond-level accuracy insufficient.

What is the speaker's view on the reliability of system measurements?

The speaker acknowledges that system measurements are often proxies and may be influenced by external factors, such as electrical signals or temperature, which can affect their reliability.

How does the speaker perceive the Dora research on software development?

The speaker views the Dora research as a scientifically valid study that provides an intellectually sound basis for reasoning about software development practices, particularly in terms of understanding the end-to-end software process.

What criticism does the speaker have regarding some of the more recent findings in the Dora report?

The speaker is skeptical about some recent findings, particularly the claim that continuous integration correlates with higher burnout in teams. They question the methodology behind these findings and argue that it may be a spurious correlation.

What is a 'spurious correlation' in the context of the Dora research?

A spurious correlation refers to a situation where two variables appear to be related, but their relationship is not causal. The speaker suggests that the link between continuous integration and negative outcomes may be coincidental rather than predictive.

How does the speaker define 'strict prediction' in research?

The speaker defines strict prediction as requiring longitudinal data and an a priori theoretical reason for the relationship. In contrast, the Dora findings are based on cross-sectional data, which limits their predictive power.

Why does the speaker think the Dora report's findings on continuous integration are problematic?

The speaker argues that there is no theoretical basis to link continuous integration directly to organizational outcomes, making it an unreliable predictor. They suggest the observed correlation is more likely due to factors like increased complexity or coordination costs in teams.