0204: Coverage measurement
Summary
TLDRThis lesson delves into test-driven development's coverage measurement, emphasizing the importance of thorough testing beyond just general cases to include edge cases. It introduces different levels of code coverage, from simple method execution (S0) to comprehensive pathway testing (C2), suggesting that while more tests are better, the real question is how thorough the testing is. The instructor highlights the balance between testing enough and over-testing, and the value of various coverage metrics in ensuring software correctness.
Takeaways
- 🔍 Coverage measurement is a key aspect of test-driven development to ensure the thoroughness of tests.
- 🛠️ Different levels of coverage provide varying degrees of confidence in the correctness of the code.
- 🤔 The question 'how much testing is enough?' is addressed by considering the balance between testing and shipping software.
- 📈 A quantitative evaluation can be made by comparing lines of test code to lines of code in the application.
- 📊 Having more tests than lines of code is not unreasonable, especially in production-level systems.
- 🧐 The focus should be on the thoroughness of testing rather than just the quantity of test code.
- 📋 Basic coverage measurements include S0 (every method is called) and S1 (every method is called from every call site).
- 🔄 C1 coverage involves executing every branch in both directions to test all possible pathways.
- 🌐 C2 coverage is about testing every possible combination of pathways through a method, which is challenging and controversial.
- 🚀 The lecture will continue with more on test-driven development in the next video.
Q & A
What is the primary focus of the lesson in this script?
-The lesson focuses on coverage measurement within the context of test-driven development, exploring different levels of coverage to evaluate how thoroughly code is being tested.
What is the basic idea behind code coverage in testing?
-Code coverage refers to the extent to which tests cover the code base. Higher coverage indicates that more parts of the code, including edge cases and different pathways, have been tested, which increases confidence in the correctness of the code.
Why is it not ideal to keep testing until the software is ready for release?
-Continuously testing until release may indicate a lack of methodical planning in the testing process, leading to ineffective or redundant testing. It's better to structure tests in a methodical manner for optimal coverage.
Why might measuring the number of test cases against lines of code be insufficient?
-Simply having more tests than lines of code does not necessarily indicate thoroughness, as the tests could be poorly written or fail to cover critical parts of the code. It's more important to consider the quality and scope of the tests.
What is S0 coverage in testing?
-S0 coverage indicates that every method in the code has been called at least once, though it doesn't guarantee that all possible code paths have been exercised or that edge cases are covered.
How does S1 coverage improve upon S0 coverage?
-S1 coverage ensures that every method is called from every possible call site, allowing for a more comprehensive evaluation of the function's behavior in different contexts, though it may still neglect fringe or edge cases.
What does C1 coverage ensure in unit testing?
-C1 coverage ensures that every statement in a method is executed at least once, including both branches of conditional statements. This provides a more thorough test than just calling the method or function.
What does C2 coverage aim to achieve in testing?
-C2 coverage aims to test every possible combination of paths through the code, requiring a large number of test inputs to ensure that every potential pathway is covered. However, it is complex and sometimes debated in terms of its value.
Why is C2 coverage considered difficult to achieve?
-C2 coverage is difficult because it requires testing every possible pathway through a function, which often involves a huge number of test cases. This makes it complex and resource-intensive, and there is debate about how valuable it is in practice.
What is the overall purpose of measuring coverage in test-driven development?
-The purpose of measuring coverage is to ensure that tests thoroughly exercise the code, increasing confidence in its correctness and stability. Higher coverage, especially when it includes edge cases and different pathways, leads to better software reliability.
Outlines
📏 Coverage Measurement in Test-Driven Development
This paragraph introduces the concept of coverage measurement in the context of test-driven development. It explains that tests can vary in their level of coverage, from general cases to edge cases. The importance of thorough testing is emphasized to ensure that the code is exercised to its fullest potential, leading to more confidence in the correctness of the code. The discussion also touches on the question of how much testing is enough, suggesting that having more tests than lines of code is not unreasonable, especially in production-level systems where failure implications are significant. The paragraph outlines different levels of coverage, starting from S0, which is simply calling every method, to S1, which involves calling every method from every call site. It also introduces the idea of executing every statement within a function and considering both directions of every branch (C1 coverage), setting the stage for a deeper discussion on test coverage in subsequent lessons.
🔍 Advanced Coverage Measurement and Its Value
This paragraph delves into more advanced forms of coverage measurement, such as considering every possible combination of pathways through a method or function. It discusses the challenge of creating test cases that can execute every path, which often requires a multitude of inputs to direct the code down different pathways. The paragraph also introduces the concept of C2 coverage, which is very difficult to achieve and is a subject of debate within the software engineering community regarding its value. The speaker acknowledges the complexity of achieving comprehensive test coverage and suggests that the next video will continue the discussion on test-driven development, implying a progression towards more practical applications of these concepts.
Mindmap
Keywords
💡Test Driven Development (TDD)
💡Coverage Measurement
💡General Case
💡Edge Cases
💡Confidence
💡Lines of Test/Code Ratio
💡Thoroughness
💡S0 Coverage
💡S1 Coverage
💡C1 Coverage
💡C2 Coverage
Highlights
Introduction to coverage measurement as part of test-driven development.
Explanation of different levels of test coverage, such as testing general cases versus edge cases.
Importance of exercising all pathways in code to ensure comprehensive testing.
Mention of Armando Fox and David Patterson's textbook as a reference for software testing concepts.
The idea that simply testing until software is ready to ship is not a methodical approach.
Quantitative evaluation: ratio of lines of test code to lines of application code.
Clarification that having more tests than lines of code is common in production-level systems.
Discussion of the limitations of just having a lot of test code, emphasizing thoroughness over quantity.
Introduction of different coverage measurements, including S0, S1, and C1.
S0 coverage ensures that every method in the application is called.
S1 coverage: calling every method from every call site, providing better insight than S0.
C1 coverage ensures every statement and every branch is tested, improving test accuracy.
C2 coverage: the most complex, requiring tests that cover every possible pathway through the code.
Challenges with C2 coverage, as it requires many inputs and may not always be practical.
The next step in the lesson is to continue with test-driven development practices.
Transcripts
Howdy, and welcome back.
So the next topic, uh, within this short lesson on test
driven development is going to be coverage measurement.
If we think about kind of crafting, uh, tests, and you
could think about doing this by hand, there's different
levels of coverage that those tests may come up with.
You might come up with tests that only exercise,
say the general case and completely disregard
fringe cases or edge cases.
This would be essentially testing the method once
and not considering all of the pathways through it.
Uh, what we'll see here is that with different level
of coverage comes to us, um, essentially more confidence
that, uh, the tests are indeed exercising the code to the
fullest ability of their potential, and that the results
are going to be rather, um, informative to us and will be-,
have that much, uh, stronger confidence in that the code
that we've written is correct.
So how much testing is enough might be our first question.
Uh, this is again from Armando Fox, and I believe
it's David Patterson's, uh, textbook, uh, kind of slide
I stole for my software engineering lecture slide.
So how much testing is enough?
Well, it's bad if we keep testing until we are ready
to ship the software.
That means we're, y'know, obviously going, not
going about the testing in a methodical manner.
It could be that we're writing our test cases towards the
end, so just continuously and mindlessly perhaps testing
until we're ready to release, uh, that that's probably not
the best way to go about this.
So one k-, more quantitative perhaps, um, evaluation of
this might be to take the lines of test and divide
it by the lines of code written in your application.
Um, having more tests than lines of code is
not, um, unreasonable.
It frequently happens.
Uh, though this occurs much more frequently in production
level systems specifically.
Uhm, y'know, those where if it does fail, the
implications are huge.
So a better question perhaps to ask than, y'know, coming
up with these metrics that are fractions and not
necessarily as informative.
Because if you just have a bunch of test code, it doesn't really
communicate how thorough it is.
It could all be garbage.
So the better question, here's going to be, how
thorough is my testing?
We'll see here that there's a number of different
cover measurements, uh, to help us describe.
So the basics, I'm going to just focus on these, uh, five.
I'm going to briefly comment on, uh, what they are and
we'll kind of leave it at that.
This is an introductory course.
It's not a software engineering course.
I'm not expecting to get too into this, but I would
like to draw your attention that there are different
coverages, um, that we can write our unit test to.
Obviously, uh, y'know, the more coverage, the better.
S0 is going to be essentially communicating that
every method within our application has been called.
And this is simply just calling it, uh, we're not necessarily,
um, calling it from every location within our application.
It's simply just being called and we're evaluating.
Um, whether or not that call succeeds in perhaps against
some basic test suite.
So notice here if we're just kind of testing it
independently, that's not too informative.
How did we, how did we get to that point, um, could,
y'know, have impact in terms of the actual errors?
So, uh, S1's a little bit better.
It's calling every method from every call site.
So every area where, uh, of that particular call is made
to a function, we go ahead and execute that line of code.
That can provide a, uh, more, a-a better understanding perhaps
as to, uh, whether or not we've successfully implemented, um,
that function because it's being called from multiple ver
multiple different context.
So if that seems to work, and that could be just kind
of looking at the general cases and maybe we're,
y'know at this point, uh, neglecting the fringe cases.
So, what can, what we're kind of interested in then
is then looking at the actual composition of the, uh, of the
function itself and writing unit tests that execute
every statement within, say that particular function.
We could take this a step further, and we can execute
every branch in both directions.
So coming up with very, uh, involved test suites
that essentially has input combinations that exercise not
only every statement presenting within, say a particular
method, like if x, do would-, could be executing that if
statement a single time.
The C1 coverage says we're going to execute every statement
and for every branch we're going to consider, we're
going to have code that's going to run us through both
the if and the else pathway.
Now we can take this to an even, uh, bigger stream is looking at.
Every combination of every pathway through a particular
method or function.
And exerci coming up with test cases that exercise, um, that,
and again, these aren't, it's not like a single test case
is going to be able to hit every path in you our code.
We're going to need to come up with a multiple, u-uh y'know
a huge number of inputs in this case so that we could
execute, uh, every path with each input, perhaps directing
us down one pathway that another took a different routing.
This last coverage C2 is very difficult and there is some
disagreement in the software engineering community, how
valuable it actually is.
So we'll go ahead and go forward with our test driven,
uh, development, uh, section of this lesson in the next video.
function.
And exerci-, coming up with test cases that exercise, um,
that, and again, these aren't, it's not like a single test
case is going to be able to hit every path in you our code.
We're going to need to come up with a multiple, u-uh y'know
a huge number of inputs in this case so that we could
execute, uh, every path with each input, perhaps directing
us down one pathway that another took a different routing.
This last coverage C2 is very difficult and there is some
disagreement in the software engineering community, how
valuable it actually is.
So we'll go ahead and go forward with our test driven,
uh, development, uh, section of this lesson in the next video.
path in you-, our code.
We're going to need to come up with a multiple, u-uh y'know
a huge number of inputs in this case so that we could
execute, uh, every path with each input, perhaps directing
us down one pathway that another took a different routing.
This last coverage C2 is very difficult and there is some
disagreement in the software engineering community, how
valuable it actually is.
So we'll go ahead and go forward with our test driven,
uh, development, uh, section of this lesson in the next video.
multiple, u-uh, y'know a huge number of inputs in this case
so that we could execute, uh, every path with each
input, perhaps directing us down one pathway that another
took a different routing.
This last coverage C2 is very difficult and there is some
disagreement in the software engineering community, how
valuable it actually is.
So we'll go ahead and go forward with our test driven,
uh, development, uh, section of this lesson in the next video.
next video.
関連動画をさらに表示
CH05.L07 . White Box techniques
ISTQB FOUNDATION 4.0 | Tutorial 35 | Branch Testing & Branch Coverage | Test Case Techniques | CTFL
L16 Unit Testing
ISTQB FOUNDATION 4.0 | Tutorial 34 | Statement Testing | Statement Coverage | Test Techniques | CTFL
Branch Coverage - Georgia Tech - Software Development Process
CH04. L01. Test development process
5.0 / 5 (0 votes)