Test Desiderata 5/12: Tests Should be Fast

Summary

In this video, Kelly and Sutton discuss the critical importance of test speed in software development. They argue that fast tests are vital for maintaining developer focus and mental flow, with delays of even a second or more disrupting concentration. Sutton critiques current test runners, which prioritize throughput over latency, suggesting a shift toward fast feedback—especially when tests fail. By rethinking test frameworks to prioritize immediate feedback, developers can stay engaged and productive, ensuring better efficiency in the development process.

Takeaways

• 😀 Fast tests are essential to keep developers in a productive 'flow' state, minimizing distractions and maintaining focus.
• 😀 Tests that run in under a second are ideal because they help developers stay engaged with the task without interruptions.
• 😀 When test execution exceeds 10 seconds, developers tend to lose focus, leading to mental distractions and reduced productivity.
• 😀 Slow tests pull developers out of flow, making them more prone to distraction, such as thinking about other unrelated tasks.
• 😀 There is a tradeoff between test throughput (how many tests run per minute) and latency (how quickly developers receive feedback on failures).
• 😀 Developers often value faster feedback on test failures rather than just running large numbers of tests quickly.
• 😀 Test latency is crucial for efficient debugging; getting feedback on failures quickly allows developers to make immediate corrections.
• 😀 Test runners should be optimized not just for throughput but also for low latency, ensuring developers get quick feedback when a test fails.
• 😀 Prioritizing the tests that are most important or critical can help reduce the time spent waiting for feedback, focusing resources where they matter most.
• 😀 Optimizing test frameworks for faster execution and reliable results can significantly improve the development process and productivity.
• 😀 Deterministic tests, which always produce the same result with the same inputs, are vital to ensure consistency and reliability during testing.

Q & A

Why is test speed important in software development?

Test speed is crucial because it directly impacts developer productivity. Fast tests allow developers to receive quick feedback, enabling them to stay in 'flow' and continue working without significant distractions or delays.

What psychological effect does slow test feedback have on developers?

Slow tests, especially those that take longer than a second, break the developer's focus and mental flow. This can lead to disengagement, distraction, or even lost time as the developer's attention shifts away from the task at hand.

How do tests that run in under a second differ from those that take longer?

Tests that run in under a second provide immediate feedback, keeping developers engaged and focused. Tests that take longer than a second, particularly those in the 10-second or minute range, interrupt this flow, leading to a less productive development environment.

What does the term 'flow' refer to in the context of software development?

In software development, 'flow' refers to a state of deep concentration and focus where a developer is fully immersed in their task. Fast test feedback helps maintain this state, while slow tests can disrupt it.

How do modern test runners prioritize throughput over latency?

Many current test runners are designed to maximize throughput—running as many tests as possible in a given time—by distributing the workload across multiple systems. This can result in slower feedback for individual tests, particularly when the initial test feedback takes several seconds or more.

What is the main issue with prioritizing throughput over latency in test runners?

The main issue is that developers often don't see the results of failing tests quickly enough. The longer it takes to get feedback, the more likely the developer's focus and energy will be disrupted, hindering productivity.

What is meant by 'latency' in the context of test runners?

Latency refers to the time it takes for a test to run and provide feedback, especially feedback about failure. Low latency is desirable because it provides quicker information on whether a test has passed or failed.

What is suggested to improve test runners and their frameworks?

The suggestion is to prioritize faster feedback for failing tests by improving test runners to reduce latency. By quickly notifying developers when a test fails, they can maintain flow and direct their energy toward fixing the issue as soon as possible.

What would be the benefit of getting feedback on failing tests within the first second?

Getting feedback on failing tests within the first second allows developers to address issues promptly while staying in their productive state of flow. This would prevent cognitive disengagement and maintain momentum in development.

Why is it not always necessary to have tests that run in under a second?

Tests that run in under a second are ideal, but not always feasible for all types of tests. However, minimizing test run time and ensuring quick failure detection is essential for keeping developers focused and reducing distractions, especially for frequently run tests.