L17 Integration Testing

Summary

This tutorial by Phil Copeman delves into integration testing, emphasizing its importance post-unit testing to ensure components work harmoniously as per high-level design. It focuses on testing component interfaces and interactions, avoiding common pitfalls like skipping to system testing or lacking traceability. The tutorial illustrates the process with a vending machine example, highlighting the need to check not just the final output but the entire sequence of actions. It also discusses the role of message dictionaries in automotive diagnostics, emphasizing the necessity of testing various message types and values to validate system behavior against design specifications.

Takeaways

• 🔍 Integration testing is conducted after unit testing to see how various pieces of the system work together.
• 🏗️ It traces back to the architectural and high-level design work, ensuring components not only function internally but also interact as intended.
• 🔄 The primary focus of integration testing is on the interfaces between components, assessing if they handle input sequences and interactions correctly.
• 📈 Integration testing aims to confirm that modules align with the high-level design, especially in terms of sequence diagrams.
• 🔄 Avoid the anti-pattern of skipping straight to system testing after unit tests, as this might miss subtle interaction issues.
• 🔗 Ensure traceability from integration tests to the high-level design to maintain the connection between testing and design intent.
• 🚫 Integration tests should not be pass/fail based on system function alone but should focus on the correctness of interfaces and interactions.
• 🛠️ A simple integration test example involves a vending machine sequence diagram, where the test checks if the system correctly processes a coin insertion and subsequent actions.
• 📊 Integration testing should cover all sequence diagrams, testing nominal behaviors, missing inputs, false preconditions, and invalid sequencing.
• 📝 High-level designs often include interface descriptions, such as message dictionaries, which should be tested for message structure, field values, and exception handling.
• 🔑 Best practices in integration testing emphasize the importance of tracing tests to the high-level design and covering all interactions to ensure the system works as intended.

Q & A

What is the primary purpose of integration testing?

The primary purpose of integration testing is to ensure that the various pieces of a software system work together as intended after individual components have been unit tested. It focuses on the interactions between components and their adherence to the high-level design.

Why is it important to conduct integration testing after unit testing?

Integration testing is important because unit testing alone cannot identify interaction problems between components. It ensures that the components, which have been individually tested, function correctly when combined and that the overall system matches the high-level design.

What are the potential anti-patterns or pitfalls of integration testing mentioned in the script?

The script mentions three anti-patterns: skipping straight to system testing after unit testing without conducting integration tests, lacking traceability from integration tests to the high-level design, and basing integration test pass criteria on system functionality rather than on the proper functioning of interfaces.

How does integration testing relate to high-level design?

Integration testing is intended to verify that the modules of a system match the high-level design, particularly focusing on the sequence diagrams and ensuring that the system behaves as the design specifies.

What is a sequence diagram and how does it relate to integration testing?

A sequence diagram is a type of diagram in the high-level design that shows the interactions between objects or components in a system over time. Integration testing uses these diagrams to check that all inputs result in correct outputs and that every component interface is exercised as expected.

What should be the focus of an integration test for a vending machine as described in the script?

The focus of an integration test for a vending machine should be to exercise a specific sequence diagram, ensuring that all inputs lead to correct outputs, every component interface is tested with relevant values, and the sequence of actions matches the expected behavior as per the high-level design.

Why is it incorrect to base the pass/fail criteria of an integration test solely on the final output?

Basing the pass/fail criteria solely on the final output is incorrect because it overlooks the importance of the intermediate results and the sequence of actions. Integration testing should verify that all arcs appear in the expected sequence and that all side effects and timings happen as expected according to the high-level design.

What is an example of an interface description used in integration testing?

An example of an interface description is the OBD2 parameter ID message dictionary, which defines automotive operational parameters and their associated data structures or fields in network packets, allowing for the testing of message structures, values, and exception handling.

How should integration testing be approached in terms of best practices?

Integration testing best practices involve focusing on the interaction of components, tracing tests to the high-level design, exercising all arcs on every sequence diagram, and covering all modules, network interfaces, message types, and data fields to ensure the system works as intended.

What are the two main pitfalls of relying solely on system testing instead of conducting integration testing?

The two main pitfalls are missing system integration edge cases where the system may appear to work but the internal logic is incorrect, and the difficulty of exercising off-nominal sequence diagrams at a system level, which can lead to overlooking specific situations that the system will not handle as intended.

Why is traceability from integration tests to the high-level design important?

Traceability is important because it ensures that the integration tests are based on a clear reference point, allowing testers to verify whether the system is behaving as the high-level design specifies. Without traceability, there is no way to know if the system is functioning as intended.

Outlines

00:00

🔍 Understanding Integration Testing

This paragraph introduces the concept of integration testing as a process that occurs after unit testing. It emphasizes the importance of ensuring that individual components, which have been verified through unit tests, work together as intended. The focus is on the interfaces between components and whether they can handle all types of data and interactions as specified in the high-level design. The paragraph also highlights common anti-patterns, such as skipping integration testing and basing pass criteria on system functions rather than interfaces. An example of a simple integration test for a vending machine is provided to illustrate the process of verifying sequence diagrams and ensuring that the software implementation matches the high-level design.

05:02

🛠️ Integration Testing Best Practices and Pitfalls

The second paragraph delves into best practices for integration testing, such as tracing tests back to the high-level design and ensuring that all sequence diagrams are covered. It discusses the importance of testing message structures, including a range of valid and invalid values, and handling exceptions like bad checksums or sequence numbers. The paragraph also warns about the pitfalls of relying solely on system testing, which can miss critical system integration edge cases and fail to exercise off-nominal sequence diagrams. The goal is to stress the interactions among components and ensure that the system works not only at a functional level but also adheres to the high-level design.

10:03

📝 The Importance of Traceability in Integration Testing

The final paragraph underscores the significance of traceability in integration testing. It points out that without a high-level design to reference, there is no clear way to determine if the system is behaving as intended. Skipping integration testing can lead to a lack of understanding of whether the system's internal logic aligns with the high-level design. The paragraph stresses the necessity of integration testing to ensure that the system's components interact correctly and that the overall system functions as designed.

Keywords

💡Integration Testing

Integration Testing is a type of software testing that occurs after unit testing, where individual units or components of a program have been tested for their functionality. The purpose of integration testing, as explained in the video, is to verify that these components work together as expected. It is a crucial step in the development process to ensure that the modules match the high-level design and that the system behaves as intended. The video uses the analogy of a vending machine to illustrate how integration testing checks the interaction between components in response to a sequence of events, such as inserting a coin.

💡Unit Testing

Unit Testing is the process of testing individual components or units of a software application to determine if they function correctly in isolation. In the context of the video, unit testing is mentioned as a prerequisite to integration testing. It ensures that each part of the software is working as intended before they are combined and tested together for interactions and integration issues.

💡High-Level Design (HLD)

High-Level Design (HLD) refers to the architectural and design aspects of a system that are developed before the actual coding begins. It includes the sequence diagrams and interface descriptions that guide the development and testing process. The video emphasizes the importance of tracing integration tests back to the HLD to ensure that the software implementation matches the intended design.

💡Sequence Diagram

A Sequence Diagram is a type of UML diagram that illustrates the interactions between objects or components in a system over time. In the video, sequence diagrams are used to show the expected sequence of events in a system, such as the steps a vending machine takes when a coin is inserted. Integration testing uses these diagrams to check if the system behaves as expected in response to specific actions.

💡Component Interfaces

Component Interfaces are the points of interaction between different components of a system. The video explains that integration testing focuses on these interfaces to ensure that data and control flow as intended between components. This is essential for identifying issues that unit testing might not catch, such as incorrect data handling or sequence errors.

💡Anti-Patterns

Anti-Patterns in the context of software development refer to common responses to a problem that are typically ineffective and/or counterproductive. The video mentions several anti-patterns in integration testing, such as skipping to system testing without proper integration testing, which can lead to subtle interaction problems being overlooked.

💡Traceability

Traceability in software testing is the ability to track the relationship between the requirements, design, and code. The video points out that without traceability from integration tests to the high-level design, it is difficult to ensure that the tests are verifying the correct aspects of the system and that the system matches the intended design.

💡System Function

System Function refers to the overall functionality of a system. The video warns against using system function as the pass criteria for integration tests, as this overlooks the importance of testing the intermediate results and interactions between components. Instead, the focus should be on the correct functioning of interfaces and sequence diagrams.

💡Message Dictionary

A Message Dictionary is a collection of definitions for messages that are used in a system, often in the context of network communications or data structures. The video uses the example of an OBD2 parameter ID message dictionary to explain how integration testing should cover the message structure, including valid and invalid values, to ensure proper message handling.

💡Validation Test Suite

A Validation Test Suite is a set of tests designed to verify that a system meets certain criteria or behaves as expected under various conditions. The video mentions that components often come with a validation test suite to ensure that all types of messages and interactions are supported correctly, which is an essential part of integration testing.

💡Best Practices

Best Practices are the most effective methods recommended for a particular activity or process. In the context of integration testing, the video outlines best practices such as focusing on the interaction of components, tracing tests to the high-level design, and ensuring comprehensive coverage of all sequence diagrams and interfaces.

Highlights

Integration testing is a process that occurs after unit testing to ensure that various pieces of a software system work together as intended.

The purpose of integration testing is to exercise all component interfaces and verify that they handle input sequences and interactions correctly according to the high-level design.

Integration testing should not duplicate unit testing efforts but focus on interaction problems that unit testing cannot find.

Skipping integration testing and moving straight to system testing can lead to missing subtle interaction problems.

Traceability from integration tests to the high-level design is crucial for ensuring the implementation matches the intended design.

Integration testing pass criteria should be based on interfaces, not just system functionality.

A simple integration test example involves a vending machine sequence diagram, where the test checks if the machine handles a coin insertion correctly.

Integration tests should monitor the cascade of actions in a sequence diagram to ensure they occur as expected.

The pass or fail of an integration test is determined by whether the system behaves according to the sequence diagrams, not just the final output.

Integration testing coverage includes checking all arcs on sequence diagrams, testing nominal behaviors, and handling missing inputs or preconditions.

Integration tests should also verify the correct implementation of side effects and timing within the system.

High-level designs often include interface descriptions, such as message dictionaries, which should be tested in integration.

Integration testing should exercise message structures, testing a range of values, valid and invalid fields, and message types.

Components should be accompanied by a validation test suite to ensure proper support for different message types.

Best practices in integration testing involve concentrating on component interactions and ensuring they satisfy high-level design aspects.

Pitfalls of system testing alone include missing system integration edge cases and difficulty in exercising off-nominal sequence diagrams.

Skipping high-level design traceability in integration testing leads to guessing the system's intended behavior.

Integration testing is essential to confirm that the system not only works but also operates exactly as the high-level design specifies.

Transcripts

this is phil copeman with a tutorial

on integration testing

integration testing sits part way up the

v

on the right hand side the idea of

integration testing

is after you're done with unit test and

the various pieces

have been checked out you then see how

the various pieces work together

integration test traces back to the left

hand side of the v

where the architectural and high-level

design work takes place

in integration testing several

components

are tested as a set each component has

an internal function

which has already been assessed by unit

test but also has communications

or the ability to control other

components

it's the arrows between components that

are the primary emphasis of

integration testing the idea of

integration testing is to exercise

all the component interfaces and ask

questions such as

did the input sequences lead to correct

responses

and can all types of data and all types

of interactions be handled by the

interfaces

as they're supposed to according to the

high-level design

integration testing is intended to make

sure that the modules match the

high-level design

including especially the sequence

diagrams

when you're doing integration testing

the idea is not to redo

things that have already been found in

unit testing

you assume unit testing has already

happened and instead

concentrate on the types of things unit

testing can't find

which primarily are interaction problems

the anti-patterns for integration

testing include

skipping straight to system test once

you're done with the unit test

you could just try and run the system

but you can easily miss

subtle interaction problems that result

in the system

almost working and you not knowing that

there's some sort of issue

that the implementation does not match

the intended high level design

because things mostly work and you don't

notice the difference

another anti-pattern is no traceability

from integration test to the high level

design

the point of integration test is to

compare against the high level design

a third anti pattern is that the

integration test pass criteria based on

system function

not interfaces that means if you run

an integration test and you ignore all

the interactions and just say yup sure

enough the system seems to work at the

end

that's not an integration test because

you're not paying attention to the

intermediate results

that's simply a system test and you're

not actually doing

integration testing

let's take a look at the types of things

that would happen in a very simple

integration test

the sequence diagram on the right shows

what happens when a coin is put into a

vending machine

an integration test tracing to this

sequence diagram

would in step one initialize all the

modules

in step 2 it would make sure as part of

or

just after the initialization each

precondition is satisfied

so that the sequence diagram can be

activated

in step 3 the initial action that

triggers the cascade of events in the

sequence diagram is fed to the system

in this case a coin is inserted

after that the integration test consists

of monitoring to make sure

that the cascade of actions in the

sequence diagram

actually takes place the coin in signal

is received

in four and five there's a side effect

that happens there may not be a way to

observe it but in number six

the result of that side effect is

observable

in the end the point of the integration

test is

to exercise a specific sequence diagram

you check that all inputs result in

correct outputs

every component interface is exercised

across the whole set of sequence

diagrams

with all the relevant values with all

the relevant timing and sequencing

if any of the sequence diagrams do not

behave as expected

that means the software does not

correctly implement the high level

design

and you have some sort of integration

testing failure

the pass fail criteria is whether or not

the system behaves as the sequence

diagrams

say the system should behave for

integration testing coverage

the questions you tend to ask are are

all arcs on all sequence diagrams

exercised are off nominal behaviors

tested

what happens if one of the inputs to a

sequence diagram is missing

what happens if one of the preconditions

is false does the sequence diagram

incorrectly trigger

do you have invalid sequencing do you

have extraneous outputs

and so on

it's important to realize that the point

of an integration test based on a

sequence

diagram is not simply the end result

in this example there's a sequence

diagram that shows a

vending machine that takes two coins for

a purchase

receiving a third coin and refunding it

automatically

going back to two coins inside the

machine ready for purchase

the integration test for this sets up

the machine

make sure that the machine thinks it

already has two coins and then

pops in another coin that initiates the

test

the remainder of the test is observing

the coin in signal arrives

the coin count increases and it thinks

it has a third coin

then it refunds the coin by exercising

the coin out actuator

and the coin count goes back to two

simply looking that the final coin out

is two

is not how you determine pass fail for

this integration test

this integration test only succeeds if

it notices it has an extra coin and it

refunds the extra coin instead of just

silently eating it

and then goes back to the right number

of coins only observing the final test

output would not tell you whether it

actually refunded the coin which is the

whole point of this sequence diagram

and this integration test what this

example illustrates

is that integration test is not simply a

pass on the final output

but rather did all the arcs appear in

the expected sequence

did all the timings happen as you

expected did all the side effects happen

as you expected

in other words it is not simply that the

pieces manage to work together more or

less

it's that they manage to work together

exactly as they're supposed to according

to the high level design

in addition to sequence diagrams it is

common for high level designs

to also have some sort of interface

description

many interfaces look like messages one

way or another

here's an example of the obd2 parameter

id message dictionary

which makes automotive operational

parameters available

via diagnostic port this is a typical

sort of message dictionary in that

each message which might be an actual

network message or it might be

a data structure in memory that you can

access has a descriptor

with a categorical value saying what

kind of message is it

is this the engine speed is the engine

coolant temperature

is it the accelerator pedal position and

so on

once you know what the enum what the

categorical value is you can then

interpret the associated data in a data

structure or

fields in a network packet based on what

that identifier is

as an example if the enum says it's an

engine speed

then there might be an integer

afterwards that is in

tens of rpms or what have you

integration testing should exercise the

message structure

it should test all types of messages

a range of values inside fields valid

and invalid field values

and invalid message types it should also

test the timing and exception handling

what if there's a bad checksum

indicating a message should be

dropped what if there's a bad sequence

number on a sequence of messages

and so on the hld will have this message

dictionary which should define

all the message types formats and so on

and give you a good basis for writing

the integration tests based on the hld

it's common to see components

accompanied by a validation test suite

so you can know that all the different

types of messages

are supported properly

integration testing best practices

revolve around concentrating on the

interaction of components

integration tests should be traced to

the high level design

including exercising all the arcs on

every sequence diagram

covering every sequence diagram in the

high level design

all the modules all the network

interfaces

all the message types all the data

fields the idea

is assuming unit testing has found

everything unit testing is likely to

find

how can you additionally stress the

interactions among components

to make sure that sure the units each do

what they want to do

but when you put them together they

actually still satisfy

all the aspects of the high level design

the two main integration testing

pitfalls

first system testing alone misses system

integration edge cases

sometimes a misbehaving system appears

to work just fine

but the internal logic isn't quite right

and there's some specific edge case

situation that it will not handle as

intended

because it was just getting lucky in the

common case

also it can be difficult to exercise off

nominal sequence diagrams at a system

level

there are some tests which are very

difficult to reproduce with a physical

system or

downright dangerous integration testing

helps you cover all the fine-grained

interactions to make sure

not only is this system working but it's

working the way you thought it was

supposed to be working

from a traceability point of view if you

skip the high level design

there's nothing to trace your

integration test to so you're just sort

of guessing

what things are supposed to do and if

you don't do integration testing

there's no way to really know whether

the system is behaving as intended

or not