L16 Unit Testing

Summary

This tutorial by Phil Copeman delves into the intricacies of unit testing, emphasizing its importance in validating software's detailed design and implementation. It distinguishes between white box and black box testing, highlighting their respective advantages and limitations. The script underscores the significance of coverage metrics like statement, branch, and MCDC coverage, and the necessity of boundary and exceptional value testing. It also advises on best practices, including the use of a unit testing framework and the importance of thorough testing to ensure robust software development.

Takeaways

• 🔍 Unit testing is a method to verify that the detailed design and implementation of a software meet the expected outcomes by testing individual components of code.
• 📐 A unit test checks various input combinations against an expected output, using an 'oracle' to determine correctness.
• 🔑 Unit tests should cover both structural (white box) and functional (black box) aspects of code to ensure comprehensive testing.
• 🐛 Catching boundary-based bugs and slight implementation imperfections is more feasible in unit testing than at the system testing level.
• 🚫 Three anti-patterns in unit testing include not performing unit tests, only testing 'happy paths', and forgetting to test missing code.
• 📦 Black box testing focuses on the software's behavior without knowledge of its implementation, which helps avoid bias but may miss internal code paths.
• 🔍 White box testing, also known as structural testing, involves examining the code to ensure all paths and conditions are tested, which can help achieve high code coverage.
• 📈 Coverage metrics are crucial for assessing the thoroughness of testing; they range from function coverage to more granular metrics like statement, branch, and MCDC coverage.
• 🛠 Achieving 100% code coverage can be challenging, especially when dealing with error handlers for unlikely events or unused code that should be removed.
• 📊 MCDC (Modified Condition/Decision Coverage) is a sophisticated coverage metric used in safety-critical applications to ensure all possible execution paths are tested.
• 📝 Boundary testing is essential for probing the limits of software behavior, including minimum and maximum values, and exceptional time-related events.
• 🛑 Exceptional value testing is vital for ensuring robust systems can handle unexpected inputs such as null pointers, undefined inputs, or unusual events like leap years.

Q & A

What is the primary purpose of unit testing?

Unit testing is a method to ensure that the detailed design and implementation of a software module meet the expected outcomes by testing individual subroutines, procedures, or methods.

What are the two main types of unit tests mentioned in the script?

The two main types of unit tests are structure or white box tests, which examine the internal workings of the code, and functionality or black box tests, which focus on the output without considering the internal implementation.

Why is it important to test both 'happy paths' and edge cases in unit testing?

Testing both 'happy paths' and edge cases is crucial because it helps uncover bugs that may arise due to slight imperfections in the code implementation, which might not be evident during regular operation.

What are the three anti-patterns for unit testing mentioned in the script?

The three anti-patterns for unit testing are: 1) Not performing unit tests at all, relying only on system-level testing, 2) Testing only the 'happy paths' and not exploring all possible scenarios, and 3) Forgetting to test missing code, such as exception handlers or special conditions that are not present in the code.

What is the difference between black box testing and white box testing?

Black box testing focuses on the software's behavior without knowledge of the internal implementation, whereas white box testing, also known as structural testing, involves knowledge of the software's actual code structure and is used to ensure all paths and conditions are tested.

Why is coverage a key concept in unit testing?

Coverage is a metric that measures the thoroughness of testing. High coverage helps ensure that all parts of the code are executed and tested, increasing the likelihood of catching bugs and ensuring the software behaves as expected under various conditions.

What is the significance of boundary testing in unit testing?

Boundary testing is important because it checks the software's behavior at the limits of its input ranges, such as minimum and maximum values, to ensure there are no unexpected results or errors at these critical points.

What is an assertion in the context of unit testing?

An assertion is a statement used within a test case to determine whether the test has passed or failed. It checks if the actual output of the tested code matches the expected outcome.

Why is it recommended to use a unit testing framework for unit testing?

Using a unit testing framework simplifies the process of creating, organizing, and running test cases. It also provides built-in functionalities for assertions and reporting, making it easier to manage and understand the results of the tests.

What are some best practices for writing effective unit tests?

Best practices for unit testing include testing every module, using a combination of white box and black box testing with granular coverage metrics, breaking tests into small, focused chunks, and ensuring both control flow and data value coverage, including boundary and exceptional value testing.

What are some pitfalls to avoid when performing unit testing?

Pitfalls to avoid in unit testing include not creating test cases, using a debugger instead of a testing framework, testing large code blobs instead of individual units, relying solely on white box testing, and neglecting peer reviews and static analysis as part of the development process.

Outlines

00:00

🔍 Introduction to Unit Testing

Phil Copeman introduces the concept of unit testing, emphasizing its importance in verifying the detailed design and implementation of software. Unit testing focuses on individual subroutines, procedures, or methods, using a low-level interface to assess their functionality. The tutorial outlines the process of testing various input combinations against an 'oracle' that defines expected outcomes. It also highlights the significance of both structure and functionality in unit tests, the potential for catching boundary-based bugs, and the three common anti-patterns to avoid in unit testing: not testing at all, only testing 'happy paths', and forgetting to test missing code. The paragraph concludes with an introduction to black box and white box testing approaches, explaining their differences and applications.

05:01

📚 Understanding White Box and Black Box Testing

This paragraph delves deeper into the two primary testing approaches: white box and black box testing. White box testing, also known as structural testing, involves examining the code's structure to ensure all paths are tested, including boundary conditions and special cases. The advantage of white box testing is the ability to achieve high structural code coverage, but it may miss testing for missing code that isn't present. On the other hand, black box testing focuses on the software's behavior without knowledge of its internal workings, making it less biased but potentially missing code paths. The importance of using both methods for comprehensive unit testing is stressed, along with the concept of coverage metrics, which measure the thoroughness of testing, from function coverage to more granular metrics like statement, branch, and MC/DC coverage.

10:01

🔬 Advanced Coverage Metrics and Boundary Testing

The paragraph discusses advanced coverage metrics, such as MC/DC (Modified Condition/Decision Coverage), which is crucial for safety-critical applications. It explains the criteria for MC/DC coverage, which ensures that every decision in the code is tested for all possible outcomes independently. Additionally, the importance of boundary testing is highlighted, which involves checking the limits of a function's behavior, including minimum and maximum values, rollovers, and exceptional time-related events. The paragraph also addresses the handling of exceptional values, such as NaN, infinity, null pointers, and unusual inputs, which are vital for robust system testing. The need for a strategy that encompasses both structural and behavioral coverage, as well as boundary and exception coverage, is emphasized for thorough unit testing.

15:02

🛠 Best Practices and Pitfalls in Unit Testing

The final paragraph provides best practices for effective unit testing, advocating for the use of a unit testing framework like CUnit to organize tests into suites and cases. It stresses the importance of high coverage, combining both white box and black box testing, and focusing on granular metrics like MC/DC or branch coverage. The paragraph also advises on the significance of data value coverage, including validating lookup tables and checking for divide by zeros. It warns against common pitfalls, such as creating test cases with bugs, using ineffective testing strategies, and relying solely on white box testing or unit testing frameworks without considering other quality assurance methods like peer reviews and static analysis.

Keywords

💡Unit Testing

Unit testing is a method of validating that individual components or units of a software function as intended. It's a fundamental aspect of the video's theme, emphasizing the importance of ensuring each part of the code works correctly before integration. The script mentions that unit testing involves testing a single subroutine or method and checking the results against expected values to confirm correctness.

💡Subroutine

A subroutine is a piece of code that performs a specific task and is used in the script to illustrate what a unit test might focus on. It's a standalone section of code that can be tested independently. The script uses the example of a subroutine that takes two inputs and produces one output to demonstrate how unit testing would be applied.

💡Oracle

In the context of the video, an oracle is a mechanism for determining the correctness of a program's output. It's a reference point or a set of expected results that a unit test checks the actual output against. The script describes the oracle as something that defines what the output 'should be' for a given function with specific inputs.

💡White Box Testing

White box testing, also known as structural testing, is a method where the tester has knowledge of the internal workings of the software. The video emphasizes its importance in unit testing, as it allows for the examination of every line of code and the paths taken within the code. The script contrasts this with black box testing, highlighting its role in achieving high structural code coverage.

💡Black Box Testing

Black box testing is an approach where the tester focuses on the software's functionality without knowledge of its internal workings. The script explains that this type of testing is based on the behavior of the software, aiming to verify the 'what' rather than the 'how'. It is used in unit testing to ensure that the software meets the specified requirements.

💡Coverage

Coverage in the script refers to the extent to which the software's code has been tested. It's a metric used to measure the thoroughness of testing. The video discusses various levels of coverage, such as function coverage, statement coverage, branch coverage, and MC/DC coverage, illustrating the importance of aiming for high coverage to ensure comprehensive testing.

💡Anti-patterns

Anti-patterns in the script are common practices in unit testing that are considered ineffective or counterproductive. The video mentions three anti-patterns: not performing unit testing, only testing 'happy paths', and forgetting to test missing code. These are highlighted as pitfalls to avoid in order to achieve effective unit testing.

💡Boundary Testing

Boundary testing is a specific type of testing that focuses on the boundaries of input values to ensure the software behaves correctly at minimum, maximum, and edge cases. The script describes boundary testing as probing the borders of behavioral changes and gives examples such as checking values just below, at, and above a minimum acceptable value.

💡Exceptional Values

Exceptional values are unusual or unexpected inputs that a program might encounter. The video script discusses the importance of testing how software handles exceptional values such as null pointers, undefined inputs, or unusual events like leap years. This type of testing ensures the software's robustness and reliability.

💡C Unit

C Unit is mentioned in the script as an example of a unit testing framework that can be used to facilitate the creation and management of unit tests. The video describes how C Unit uses a test registry, test suites, and assertions to organize and verify the correctness of tests, emphasizing the practical application of unit testing methodologies.

💡Assertions

Assertions are statements used within unit tests to check if a condition is true. If the condition is false, the assertion fails, indicating that the test did not pass. The script uses assertions as a way to determine whether the output of a unit under test matches the expected results, exemplifying their role in verifying the correctness of software behavior.

Highlights

Unit testing is a method to ensure the detailed design and implementation align with expectations.

It involves testing individual code modules like subroutines, procedures, or methods.

Unit tests use a low-level interface to check various input combinations against expected outcomes.

Results are verified against an oracle, which defines the correct output for given inputs.

Unit tests should cover both structure (white box) and functionality (black box).

Unit testing is crucial for catching boundary-based bugs and implementation imperfections.

Three anti-patterns in unit testing include not testing, only testing happy paths, and forgetting to test missing code.

Black box testing focuses on software behavior without knowledge of the detailed implementation.

White box testing, or structural testing, examines the software with knowledge of its codebase.

Coverage metrics are essential for assessing the thoroughness of testing.

Function coverage is a basic metric measuring the number of tested functions.

Statement coverage measures the percentage of executed code lines.

Branch coverage assesses whether all code paths have been tested.

MCDC (Modified Condition/Decision Coverage) is a sophisticated metric used in safety-critical applications.

Boundary testing checks the borders of behavioral changes, such as minimum and maximum values.

Exceptional value testing ensures robustness against unusual or unexpected data inputs.

Unit testing frameworks like CUnit facilitate organized and effective testing practices.

Best practices for unit testing include comprehensive coverage, use of frameworks, and small, focused test cases.

Pitfalls of unit testing include the resource-intensive nature of creating test cases and potential bugs within test cases themselves.

Unit testing should not replace peer reviews and static analysis but complement them in the development process.

Transcripts

hi this is phil copeman with a tutorial

on unit testing

we've covered requirements and design

and now we'll be talking about

unit test which is a way to check to

make sure the detailed design

and the implementation turned out as you

expected

unit testing involves testing a single

subroutine

procedure method or other standalone

chunk of code

it uses a low-level interface so you can

think of

unit as being a code module here's an

example of a subroutine that takes two

inputs and produces one output

a unit test would test several

combinations of inputs for example

a equals zero and b equals zero is one

unit test

and a equals negative one b is plus one

is a different unit test the results of

the unit test

are checked against some oracle

something that says

this is what it should be for this

function the two inputs being zero is

supposed to result in zero

and negative one and plus two are

supposed to result in plus one

if those results match the expected

values then the unit test passes

unit tests should include both structure

and functionality

structure or white box tests

functionality of black box tests and

we'll talk about both of those

unit testing is your best chance to

catch boundary based bugs

and other bugs due to slight

imperfections in how the code was

implemented

it is going to be much easier to find

those sorts of bugs

here in unit test than at the system

testing level

there are three anti-patterns for unit

testing

one anti-pattern is not doing it some

systems are only tested at the system

level

and that's a lost opportunity to find

many detailed bugs that will not

be found in most system testing but

could have been found in unit testing

another anti-pattern is testing only

happy paths

it's common to see unit tests that just

confirm what the programmer

expected to see instead of really trying

to delve down

into all the deep dark corners of the

way the code was written

a subtle but critical ansi pattern is

forgetting to test

missing code sometimes there are pieces

of code that should be there

that aren't and if you look at the code

and say well we're going to just test

based on what's in the code

you won't ever think of those sorts of

things such as exception handlers or

missing special conditions

because they're not in the code and it

didn't occur to you to test things that

you're not looking at

in the code

there are two different generic

approaches to doing testing

one is black box testing black box

testing

uses test design based on knowledge of

the behavior of the software

but not the detailed implementation it

is often called functional

or behavioral testing the idea is to

test the

what but not the how as an example

if you have a automotive cruise control

that system might

behave differently at different speeds a

black box test

might try different various speeds and

different control combinations

but would have no idea if there's

special cases inside the code that

handle things

it might hit them it might not the

advantage to a black box test

is it can be written only based on the

requirements or design

you don't need to know what's inside the

code you can just say

gee does this thing do more or less what

it's supposed to

another advantage is that sometimes

looking at the code makes you think a

certain way

and black box testing is not subject to

that particular bias

the disadvantage of black box testing is

that it can be difficult to exercise all

the code paths

because you don't really know what's

going on inside the code you just know

at the high functional level

you seem to be hitting all the high

level requirements

at the unit testing level black box

testing

looks like basing tests on the detailed

design

this means you would look at the

flowchart or state chart and design

tests

just looking at the flowchart and state

chart and confirm that the software

behaves the way the design says it

should

the other general approach to testing is

white box testing

in white box testing the designer has

knowledge of how the software is

actually written

that means you're actually looking at

the code when you design the white box

tests

white box testing is often called

structural testing

or sometimes glass box or clear box

testing

but white box testing is the preferred

term the idea is to

exercise software knowing how it is

written

back to the cruise control automotive

example a white box test would look at

all the lines of code and make sure that

all the paths in the code are exercised

and if there's a lookup table make sure

that every entry in the lookup table has

been tested

knowing exactly where the boundaries are

between

when each entry in the table would be

applied

an advantage to white box testing is

that it helps get

high structural code coverage black box

testing

struggles to hit all lines of code

because it's operating blindfolded

with whitebox testing if you miss a line

of code you can say oh there's a line of

code i need to test

let me think about how to get at that

line of code

the good news is with whitebox testing

you can get at all the lines of code

because you get to see what's going on

an important disadvantage though is that

you don't necessarily test code that's

missing

what do i mean by missing code well

let's say there's a special case you

should handle

if that special case is not in the code

at all nothing in the code will tell you

you should be testing for it

that means that black box and white box

testing both have their roles and you

should be using both of them for unit

testing

as with all testing the concept of

coverage

is key coverage is a metric for how

thorough your testing is

getting high coverage can be difficult

think of it as trying to paint a wall

painting the wide open middle pretty

straightforward takes a few roller

strokes

getting into the nooks and crannies in

the corners that's a little bit more

delicate work

takes more time takes more attention

coverage

metrics tend to reflect this the

simplest

coarsest metric for unit testing is

function coverage

how many functions in your system did

you test at all

if you never even tried to test half the

functions well

that's probably not very thorough

testing is it

once you've gotten to the point where

most or all of the functions have been

tested

at least a little a finer grain notion

of coverage is statement coverage

what fraction of the code statements

have been tested

did you actually execute every line of

code at least once

a typical metric here might be that

you've tested

95 percent of the lines of code well 95

sounds good but that means 5 of the

codes in your system

have never been executed at all and

depending on your system that could be a

problem

once you've executed all the lines of

code there are still

better types of coverage metrics

available the next one is branch

coverage

also called path coverage in branch

coverage you say all right i've executed

all the lines of code

but did i try each branch both ways this

one's a little subtle

if you have an if else statement then

100 statement coverage means you also

got 100

branch coverage because you must have

done both the if and the else

but branch coverage also includes the

notion that if you have an if without an

else did you do

both the if and did you also do the if

being false which

skips the code if you missed the

skipping the code you still might have

100

statement coverage but you'd be missing

that half of the branch coverage for

that if statement

an even finer grain notion of control

flow coverage is mcdc coverage which

we'll handle in the next slide

getting to 100 coverage can be a bit

tricky

if you have error handlers for errors

that are never supposed to happen

you can't get 100 coverage unless you

make that error happen

now you might say well it's never

supposed to happen so what's the big

deal

well the counter to that is if you have

an error handler that you've never

actually tested how do you know it's

going to work if that error actually

occurs

you need to find some test method to

exercise the handler to make sure it

really works

another issue with 100 code coverage is

dead or unused code and usually there

what you should be doing is

if that code cannot ever be executed why

is it there

probably you should take it out

an even more sophisticated type of

coverage is

mcdc coverage for white box testing

mcdc is modified condition decision

coverage it's used by do178

for critical aviation software testing

and is commonly used for other safety

critical applications where you have to

be really sure the code's always going

to work

and do exactly the right thing or else

somebody could die

mcdc coverage exercises all the ways to

reach

all the code the criteria for mcdc

coverage are that

every entry and exit point is invoked

every decision tries every possible

outcome

each condition in a decision generates

all the outcomes

and each condition in a decision is

shown to independently affect the

outcome of the decision

that's quite a lot to take in but the

idea is

in branch coverage you do the if and the

else

but in mcdc coverage if there's five

different ways the if could be true

you need to try all five different ways

and then also show the else could happen

as an example the condition clause if a

is equal to 3 or b is equal to 4

for branch coverage you'd just need to

test that it was true and that it was

false

for mcdc coverage you have to say well

if a is 3 and b is not 4

then we know a activates the true

statement

if a is not equal to 3 and b is 4 then

we know that b

can activate the true statement and if

both a is not equal to 3

and b is not equal to 4 we know that the

else can be

activated even if it's just a fall

through now

it seems like almost everything has to

be tried you do not have to try

a is 3 and b is 4 because that's already

been covered by the first two cases

if you're not quite sure go look at the

list at the top

of the four decision criteria and

convince yourself the first three test

cases check all the boxes and the fourth

one is not needed

for simple ants and ors you generally

need one more test

than the number of terms in the

condition clause for two terms you need

three tests

for five terms you need six tests and so

on

if you have more complicated expressions

there's a truth table technique that can

help and you may need trial and error to

find

all the different variations the next

item in the playlist

is a video from another author which

works a truth table example for mcdc

coverage

there are a number of other possible

considerations when considering unit

testing coverage

a common criteria for good unit tests is

to exercise

boundaries known as boundary testing

boundary testing probes the borders of

behavioral changes

as examples if you have a minimum

acceptable value

you check one notch below the minimum

exactly at the minimum and one notch

above the minimum to make sure there's

no unexpected results

that includes minimum and maximum values

things like counter rollovers

wraparounds overflows and those sorts of

things

a special category of boundary tests

that is important in practice

are time crossings hours crossing

midnight

days crossing the end of the month leap

years leap days

years rolling over from 1999 to 2000

and so on a different type of unit

testing coverage that also matters for

robust systems

is handling exceptional values every

time there's

a floating point computation ask

yourself what happens

if a not a number or an infinity is

pulled into the equation

for pointers ask what happens if there's

a null pointer that's dereferenced

for string operations ask what happens

if there's a null string or a string

with no

terminating character other exceptional

values include

undefined inputs or invalid inputs or

unusual events

such as leap year or daylight savings

time changes

in other words exceptional value testing

is both control flow but also

often more importantly special data

values that are not the normal expected

values but which can still happen

and cause your software to misbehave

when you think you're done unit testing

you should be able to justify

your level of coverage is your software

good enough if you only get statement

coverage or do you need

mcdc coverage and did you cover all the

special cases did you cover all the

potential data issues

did you look for divide by zero and so

on

it's important to define a strategy not

only for structural and behavior

coverage

but also for boundary and exception

coverage to get good

thorough unit testing

while you can do your own unit testing

from the ground up

it's smart to use a unit testing

framework if one is available in your

development environment

c unit is a classic example framework

although it doesn't really matter which

one you use as long as it suits your

purposes

in c unit there's a test registry which

contains a number of test suites

and a test suite is a set of related

test cases

a test case is a procedure that runs one

or more executions of a module

for the purpose of testing typically

it's better if a test case has either

one case

or a number of variations on a very

narrow theme

to keep things consistent within the

test case

an assertion is a statement that

determines if a test has passed or fail

remember that just exercising the

software isn't good enough you have to

also know

that the test passed by comparing the

output values to see if they're what's

expected

as an example of a test case let's

consider you have a function that takes

two integers and returns the maximum you

would define

test maxi with the test underscore part

being there to help the framework know

that these two routines correspond to

each other

test maxi is a test case and has three

different assertions in it

cu standing for c unit assert is true

if the logical relation is true and

false if it's not

the test passes if it's true the test

fails if it's false

and here we say the max of 0 and 2

should be 2

the max of 0 negative 2 should be 0 and

the max of 2 and 2 should be 2.

in practice a c unit test looks like

long lists of cu asserts or other

similar assertions

and inside the assertion is a test that

sends inputs to the unit you're trying

to test and then checks for the expected

output

here are some best practices for unit

testing

you should unit test every module in

your system

use a high coverage combination of white

box and black box testing

ideally with a fairly granular coverage

metric such as

mcdc coverage or branch coverage

use a unit testing framework such as c

unit and break tests up into fairly

small

chunks so it's easy to understand what

each test does

and you can be sure that you did all the

things you intended to do

in terms of different types of coverage

don't forget

to not only get control flow coverage

which is what those metrics were about

but also good data value coverage such

as validating all lookup table

entries checking for divide by zeros

checking for unusual exceptional data

values and so on

there are a number of pitfalls to unit

testing creating test cases is a

development effort

it takes significant resources and guess

what test cases can also have bugs

so a failed test case could be a bug in

the code but it could also be a bug in

the test case

nonetheless it's important to do the

test cases to make sure that your code

is as thoroughly debugged as possible

before you begin

more integrated system level testing if

you've written code that is difficult to

test

it can lead to dysfunctional unit test

strategies

for example sometimes people put their

system into a breakpoint debugger

take the entire source code image and

try and do unit testing with a

breakpoint

debugger that is not an effective unit

test strategy

if that's the only way to run your

functions then you should probably

redesign to make unit test more feasible

similarly just because you're using c

unit doesn't mean you're doing unit

testing

using c unit to test a hundred thousand

line blob of code

well that is a sort of testing but

you're not going to get very good

coverage and it's not really what we

meant by unit testing

if you use only white box testing it is

doomed to succeed

and by that i mean there are tools that

will automatically generate

tests that cover all your lines of code

but those tools have no way of knowing

what the values are supposed to be

unless you feed them additional

information and they're not going to

generate code you forgot to put in

such as missing special cases finally

do not substitute unit testing for peer

reviews and static analysis

you should be doing peer reviews you

should be doing static analysis

and unit testing should come after the

code looks clean from both of those

methods