0204: Coverage measurement

Summary

This lesson delves into test-driven development's coverage measurement, emphasizing the importance of thorough testing beyond just general cases to include edge cases. It introduces different levels of code coverage, from simple method execution (S0) to comprehensive pathway testing (C2), suggesting that while more tests are better, the real question is how thorough the testing is. The instructor highlights the balance between testing enough and over-testing, and the value of various coverage metrics in ensuring software correctness.

Takeaways

• 🔍 Coverage measurement is a key aspect of test-driven development to ensure the thoroughness of tests.
• 🛠️ Different levels of coverage provide varying degrees of confidence in the correctness of the code.
• 🤔 The question 'how much testing is enough?' is addressed by considering the balance between testing and shipping software.
• 📈 A quantitative evaluation can be made by comparing lines of test code to lines of code in the application.
• 📊 Having more tests than lines of code is not unreasonable, especially in production-level systems.
• 🧐 The focus should be on the thoroughness of testing rather than just the quantity of test code.
• 📋 Basic coverage measurements include S0 (every method is called) and S1 (every method is called from every call site).
• 🔄 C1 coverage involves executing every branch in both directions to test all possible pathways.
• 🌐 C2 coverage is about testing every possible combination of pathways through a method, which is challenging and controversial.
• 🚀 The lecture will continue with more on test-driven development in the next video.

Q & A

What is the primary focus of the lesson in this script?

The lesson focuses on coverage measurement within the context of test-driven development, exploring different levels of coverage to evaluate how thoroughly code is being tested.

What is the basic idea behind code coverage in testing?

Code coverage refers to the extent to which tests cover the code base. Higher coverage indicates that more parts of the code, including edge cases and different pathways, have been tested, which increases confidence in the correctness of the code.

Why is it not ideal to keep testing until the software is ready for release?

Continuously testing until release may indicate a lack of methodical planning in the testing process, leading to ineffective or redundant testing. It's better to structure tests in a methodical manner for optimal coverage.

Why might measuring the number of test cases against lines of code be insufficient?

Simply having more tests than lines of code does not necessarily indicate thoroughness, as the tests could be poorly written or fail to cover critical parts of the code. It's more important to consider the quality and scope of the tests.

What is S0 coverage in testing?

S0 coverage indicates that every method in the code has been called at least once, though it doesn't guarantee that all possible code paths have been exercised or that edge cases are covered.

How does S1 coverage improve upon S0 coverage?

S1 coverage ensures that every method is called from every possible call site, allowing for a more comprehensive evaluation of the function's behavior in different contexts, though it may still neglect fringe or edge cases.

What does C1 coverage ensure in unit testing?

C1 coverage ensures that every statement in a method is executed at least once, including both branches of conditional statements. This provides a more thorough test than just calling the method or function.

What does C2 coverage aim to achieve in testing?

C2 coverage aims to test every possible combination of paths through the code, requiring a large number of test inputs to ensure that every potential pathway is covered. However, it is complex and sometimes debated in terms of its value.

Why is C2 coverage considered difficult to achieve?

C2 coverage is difficult because it requires testing every possible pathway through a function, which often involves a huge number of test cases. This makes it complex and resource-intensive, and there is debate about how valuable it is in practice.

What is the overall purpose of measuring coverage in test-driven development?

The purpose of measuring coverage is to ensure that tests thoroughly exercise the code, increasing confidence in its correctness and stability. Higher coverage, especially when it includes edge cases and different pathways, leads to better software reliability.

Outlines

00:00

📏 Coverage Measurement in Test-Driven Development

This paragraph introduces the concept of coverage measurement in the context of test-driven development. It explains that tests can vary in their level of coverage, from general cases to edge cases. The importance of thorough testing is emphasized to ensure that the code is exercised to its fullest potential, leading to more confidence in the correctness of the code. The discussion also touches on the question of how much testing is enough, suggesting that having more tests than lines of code is not unreasonable, especially in production-level systems where failure implications are significant. The paragraph outlines different levels of coverage, starting from S0, which is simply calling every method, to S1, which involves calling every method from every call site. It also introduces the idea of executing every statement within a function and considering both directions of every branch (C1 coverage), setting the stage for a deeper discussion on test coverage in subsequent lessons.

05:01

🔍 Advanced Coverage Measurement and Its Value

This paragraph delves into more advanced forms of coverage measurement, such as considering every possible combination of pathways through a method or function. It discusses the challenge of creating test cases that can execute every path, which often requires a multitude of inputs to direct the code down different pathways. The paragraph also introduces the concept of C2 coverage, which is very difficult to achieve and is a subject of debate within the software engineering community regarding its value. The speaker acknowledges the complexity of achieving comprehensive test coverage and suggests that the next video will continue the discussion on test-driven development, implying a progression towards more practical applications of these concepts.

Keywords

💡Test Driven Development (TDD)

Test Driven Development (TDD) is a software development approach where tests are written before the actual code. This ensures that the code is designed to be testable and allows developers to catch errors early in the development process. In the video, TDD is the overarching theme, with coverage measurement being a key aspect of this methodology.

💡Coverage Measurement

Coverage measurement refers to assessing how thoroughly the source code is tested by a given set of tests. It is a metric used to determine the effectiveness of testing in a software development process. The video emphasizes the importance of coverage measurement as a way to gauge the confidence in the correctness of the code.

💡General Case

The general case is a scenario in testing where the most common or expected inputs and conditions are considered. The video mentions that some tests might only cover the general case, ignoring edge cases, which could lead to incomplete testing and potential undiscovered bugs.

💡Edge Cases

Edge cases are scenarios that test the boundaries of the input range or unusual conditions. They are critical for uncovering bugs that may not appear under normal circumstances. The script discusses the importance of considering edge cases in testing to ensure robustness of the software.

💡Confidence

In the context of the video, confidence refers to the assurance that the software functions correctly as intended. Higher levels of test coverage contribute to greater confidence in the software's reliability and correctness.

💡Lines of Test/Code Ratio

This ratio compares the amount of test code to the amount of production code written. The video suggests that having more lines of test code than production code is not unreasonable, especially in production-level systems where failure could have significant consequences.

💡Thoroughness

Thoroughness in testing refers to the extent to which all possible scenarios are considered and tested. The video posits that thorough testing is more valuable than simply having a large quantity of tests, as it ensures comprehensive coverage of all potential code paths.

💡S0 Coverage

S0 Coverage is a basic level of testing where every method in the application is called at least once. The video uses S0 as an example of a minimal testing approach, suggesting that while it's a start, it does not provide deep insights into the functionality of the code.

💡S1 Coverage

S1 Coverage involves calling every method from every possible call site. This is a more comprehensive approach than S0, as it tests how methods behave when called from different contexts within the application, providing a better understanding of their functionality.

💡C1 Coverage

C1 Coverage is a testing level where every statement and every branch (like if-else conditions) are executed. This ensures that all code paths are tested, including both the true and false outcomes of conditional statements.

💡C2 Coverage

C2 Coverage aims to test every possible combination of code paths within a method. This is a very high level of testing that is challenging to achieve and is debated in the software engineering community for its practical value. The video mentions this as the most difficult type of coverage to achieve.

Highlights

Introduction to coverage measurement as part of test-driven development.

Explanation of different levels of test coverage, such as testing general cases versus edge cases.

Importance of exercising all pathways in code to ensure comprehensive testing.

Mention of Armando Fox and David Patterson's textbook as a reference for software testing concepts.

The idea that simply testing until software is ready to ship is not a methodical approach.

Quantitative evaluation: ratio of lines of test code to lines of application code.

Clarification that having more tests than lines of code is common in production-level systems.

Discussion of the limitations of just having a lot of test code, emphasizing thoroughness over quantity.

Introduction of different coverage measurements, including S0, S1, and C1.

S0 coverage ensures that every method in the application is called.

S1 coverage: calling every method from every call site, providing better insight than S0.

C1 coverage ensures every statement and every branch is tested, improving test accuracy.

C2 coverage: the most complex, requiring tests that cover every possible pathway through the code.

Challenges with C2 coverage, as it requires many inputs and may not always be practical.

The next step in the lesson is to continue with test-driven development practices.

Transcripts

Howdy, and welcome back.

So the next topic, uh, within this short lesson on test

driven development is going to be coverage measurement.

If we think about kind of crafting, uh, tests, and you

could think about doing this by hand, there's different

levels of coverage that those tests may come up with.

You might come up with tests that only exercise,

say the general case and completely disregard

fringe cases or edge cases.

This would be essentially testing the method once

and not considering all of the pathways through it.

Uh, what we'll see here is that with different level

of coverage comes to us, um, essentially more confidence

that, uh, the tests are indeed exercising the code to the

fullest ability of their potential, and that the results

are going to be rather, um, informative to us and will be-,

have that much, uh, stronger confidence in that the code

that we've written is correct.

So how much testing is enough might be our first question.

Uh, this is again from Armando Fox, and I believe

it's David Patterson's, uh, textbook, uh, kind of slide

I stole for my software engineering lecture slide.

So how much testing is enough?

Well, it's bad if we keep testing until we are ready

to ship the software.

That means we're, y'know, obviously going, not

going about the testing in a methodical manner.

It could be that we're writing our test cases towards the

end, so just continuously and mindlessly perhaps testing

until we're ready to release, uh, that that's probably not

the best way to go about this.

So one k-, more quantitative perhaps, um, evaluation of

this might be to take the lines of test and divide

it by the lines of code written in your application.

Um, having more tests than lines of code is

not, um, unreasonable.

It frequently happens.

Uh, though this occurs much more frequently in production

level systems specifically.

Uhm, y'know, those where if it does fail, the

implications are huge.

So a better question perhaps to ask than, y'know, coming

up with these metrics that are fractions and not

necessarily as informative.

Because if you just have a bunch of test code, it doesn't really

communicate how thorough it is.

It could all be garbage.

So the better question, here's going to be, how

thorough is my testing?

We'll see here that there's a number of different

cover measurements, uh, to help us describe.

So the basics, I'm going to just focus on these, uh, five.

I'm going to briefly comment on, uh, what they are and

we'll kind of leave it at that.

This is an introductory course.

It's not a software engineering course.

I'm not expecting to get too into this, but I would

like to draw your attention that there are different

coverages, um, that we can write our unit test to.

Obviously, uh, y'know, the more coverage, the better.

S0 is going to be essentially communicating that

every method within our application has been called.

And this is simply just calling it, uh, we're not necessarily,

um, calling it from every location within our application.

It's simply just being called and we're evaluating.

Um, whether or not that call succeeds in perhaps against

some basic test suite.

So notice here if we're just kind of testing it

independently, that's not too informative.

How did we, how did we get to that point, um, could,

y'know, have impact in terms of the actual errors?

So, uh, S1's a little bit better.

It's calling every method from every call site.

So every area where, uh, of that particular call is made

to a function, we go ahead and execute that line of code.

That can provide a, uh, more, a-a better understanding perhaps

as to, uh, whether or not we've successfully implemented, um,

that function because it's being called from multiple ver

multiple different context.

So if that seems to work, and that could be just kind

of looking at the general cases and maybe we're,

y'know at this point, uh, neglecting the fringe cases.

So, what can, what we're kind of interested in then

is then looking at the actual composition of the, uh, of the

function itself and writing unit tests that execute

every statement within, say that particular function.

We could take this a step further, and we can execute

every branch in both directions.

So coming up with very, uh, involved test suites

that essentially has input combinations that exercise not

only every statement presenting within, say a particular

method, like if x, do would-, could be executing that if

statement a single time.

The C1 coverage says we're going to execute every statement

and for every branch we're going to consider, we're

going to have code that's going to run us through both

the if and the else pathway.

Now we can take this to an even, uh, bigger stream is looking at.

Every combination of every pathway through a particular

method or function.

And exerci coming up with test cases that exercise, um, that,

and again, these aren't, it's not like a single test case

is going to be able to hit every path in you our code.

We're going to need to come up with a multiple, u-uh y'know

a huge number of inputs in this case so that we could

execute, uh, every path with each input, perhaps directing

us down one pathway that another took a different routing.

This last coverage C2 is very difficult and there is some

disagreement in the software engineering community, how

valuable it actually is.

So we'll go ahead and go forward with our test driven,

uh, development, uh, section of this lesson in the next video.

function.

And exerci-, coming up with test cases that exercise, um,

that, and again, these aren't, it's not like a single test

case is going to be able to hit every path in you our code.

We're going to need to come up with a multiple, u-uh y'know

a huge number of inputs in this case so that we could

execute, uh, every path with each input, perhaps directing

us down one pathway that another took a different routing.

This last coverage C2 is very difficult and there is some

disagreement in the software engineering community, how

valuable it actually is.

So we'll go ahead and go forward with our test driven,

uh, development, uh, section of this lesson in the next video.

path in you-, our code.

We're going to need to come up with a multiple, u-uh y'know

a huge number of inputs in this case so that we could

execute, uh, every path with each input, perhaps directing

us down one pathway that another took a different routing.

This last coverage C2 is very difficult and there is some

disagreement in the software engineering community, how

valuable it actually is.

So we'll go ahead and go forward with our test driven,

uh, development, uh, section of this lesson in the next video.

multiple, u-uh, y'know a huge number of inputs in this case

so that we could execute, uh, every path with each

input, perhaps directing us down one pathway that another

took a different routing.

This last coverage C2 is very difficult and there is some

disagreement in the software engineering community, how

valuable it actually is.

So we'll go ahead and go forward with our test driven,

uh, development, uh, section of this lesson in the next video.

next video.