CH05. L03. Test planning activities and time estimation

Summary

This script delves into the intricacies of test planning, outlining 10 key activities. It emphasizes identifying testing scope, risks, and objectives, choosing between proactive and reactive approaches. The discussion covers integrating test activities within the software lifecycle, deciding what to test, assigning resources, and setting schedules. It also highlights the importance of establishing entry and exit criteria, monitoring, and controlling the testing process. The script further explores test estimation techniques and the influence of product characteristics, development processes, and testing outcomes on the testing effort. Finally, it touches on various test approaches, including analytical, stochastic, methodical, and dynamic heuristic methods, underlining the context-dependent nature of testing.

Takeaways

• 🔍 Identify the scope of testing, the risks involved, objectives, and the approach (proactive or reactive) as the first step in test planning.
• 🔄 Integrate and coordinate test activities within the software life cycle, ensuring each life cycle activity has a corresponding testing activity.
• 📝 Decide on the parts to test, roles needed, and how to evaluate test results during the planning stage.
• 🗓 Set a detailed schedule for each task, including start and end times, participants, and duration.
• 👥 Assign resources to each activity, identifying the level of detail and selecting necessary metrics for monitoring and control.
• 👀 Monitor and control the test process to ensure execution aligns with the plan and take immediate action to fix any deviations.
• 📋 Specify the level of detail in test procedures to guide the preparation and execution of testing activities.
• 🚦 Define entry criteria for when to start testing, including the readiness of the test environment, tools, data, and code.
• 🏁 Establish exit criteria to determine when to stop testing, such as coverage targets, defect density, or risk analysis.
• 📊 Test effort estimation should be based on approaches like matrix-based or expert-based to ensure fairness and accuracy.
• ⚖️ Test effort is influenced by product characteristics, development process, and testing outcomes, including defect trends and rework required.

Q & A

What is the first step in test planning activities?

The first step is to identify the scope of testing, assess the risks, define the testing objectives, and decide on the test planning approach (either proactive or reactive).

How are test activities integrated into the software life cycle?

Test activities are integrated and coordinated within the software life cycle by aligning each life cycle activity with its corresponding testing activity in the V-model.

What is meant by 'monitor and control' in the test planning process?

'Monitor and control' refers to the follow-up process where the execution is monitored to ensure it matches the plan, and immediate actions are taken to correct any deviations.

What are entry criteria in the context of test planning?

Entry criteria specify the conditions that must be met before testing can start, such as having a ready test environment, necessary tools, and test data.

What are exit criteria in test planning?

Exit criteria define the conditions under which testing can be concluded, such as achieving a certain level of code coverage, completing a specified number of test cases, or resolving critical defects.

How is the test effort estimated?

Test effort is estimated using approaches like matrix-based estimation, which relies on data from similar projects, and expert-based estimation, which depends on the experience of task owners.

What factors influence the testing effort?

Factors influencing the testing effort include the characteristics of the product, the development process, and the outcomes of testing, such as the number of defects found and the required rework.

What is the analytical approach in test strategy?

The analytical approach, also known as risk-based testing, focuses the testing effort on modules with the greatest risk by analyzing the system to identify high-risk areas.

What is stochastic testing?

Stochastic testing, or random testing, uses statistical data about failure rates to measure the system's reliability by inputting multiple inputs in a specific sequence and checking the system's response.

What is the methodical approach in testing?

The methodical approach deals with failures by using techniques like error guessing and fault attacks, relying on the test engineer's experience to determine the testing method.

Outlines

00:00

🔍 Overview of Test Planning Activities

This paragraph provides a detailed overview of the test planning activities. It begins by discussing the importance of identifying the scope, risks, and objectives of testing. The paragraph outlines the integration of testing within the software life cycle, the selection of proactive or reactive approaches, and the scheduling of tasks. It also emphasizes the significance of assigning resources, defining entry and exit criteria, and ensuring the availability of necessary tools and environments before testing begins. The discussion concludes with the importance of monitoring and control, highlighting the need for immediate action to correct deviations during testing.

05:03

🚦 Entry and Exit Criteria in Testing

This paragraph delves into the concepts of entry and exit criteria in the testing process. Entry criteria refer to the conditions that must be met before testing can begin, such as the availability of a ready test environment, tools, and test data. The paragraph stresses the importance of preparing these elements in advance to avoid delays. Exit criteria determine when testing should be concluded, based on factors like code coverage, defect density, and the completion of specific test levels. It discusses the need to focus on achieving testing objectives and managing risks before concluding the testing process.

10:05

🧠 Approaches to Test Estimation and Strategies

This paragraph focuses on the test estimation process and the factors that influence it. It describes how test leaders estimate the effort required for testing tasks using methods like matrix-based and expert-based approaches. The estimation considers the characteristics of the product, development process, and testing outcomes, such as the number of defects found and the amount of rework required. The paragraph also touches on the test strategy and approach, highlighting the importance of context-dependent testing and selecting the appropriate techniques based on the project’s requirements.

⚖️ Different Testing Approaches and Their Applications

This paragraph explores various testing approaches and their applications in specific contexts. It introduces seven common testing approaches, including analytical, stochastic, methodical, process-compliant, dynamic and heuristic, consultative, and regression averse. The paragraph explains each approach, such as the analytical approach focusing on risk-based testing, stochastic testing using statistical models, and methodical approaches relying on error guessing and fault attacks. It also covers the importance of selecting the right approach based on factors like risk, safety, available resources, and the specific needs of the project.

Keywords

💡Test Planning

Test planning refers to the process of defining the scope, approach, resources, and schedule of intended test activities. It involves identifying risks, setting objectives, and integrating test activities within the software development life cycle. For example, deciding which parts of the software need testing and determining the test schedule.

💡Proactive and Reactive Approaches

Proactive and reactive approaches refer to different strategies in test planning. Proactive planning involves preparing for potential issues in advance, while reactive planning addresses issues as they arise. The script mentions deciding which approach to use as part of identifying the testing scope and risks.

💡Test Activities Integration

Integrating and coordinating test activities within the software life cycle ensures that testing is aligned with development. Each development activity has a corresponding test activity, particularly in models like the V-model. This integration helps in scheduling and assigning resources for each test activity.

💡Entry Criteria

Entry criteria are the conditions that must be met before testing can begin. They include the availability of the test environment, tools, and data. The script explains that ensuring these criteria are met prevents delays and inefficiencies during the testing process.

💡Exit Criteria

Exit criteria define when to stop testing, based on specific goals like achieving a certain coverage or defect density. They help in determining the completion of testing phases and ensuring that the product meets the required quality standards before release.

💡Test Estimation

Test estimation involves predicting the effort, time, and resources required for testing activities. It can be based on matrix methods using historical data or expert judgment. Accurate estimation ensures fair task distribution and realistic scheduling.

💡Risk-Based Testing

Risk-based testing focuses on identifying and testing the most critical and high-risk areas of the software first. This analytical approach directs the testing effort towards modules that have the greatest impact on the system's performance and reliability.

💡Test Design Techniques

Test design techniques are methods used to create test cases and scenarios. These can include analytical, statistical, and experience-based approaches, depending on the context and objectives of the testing. Choosing the right technique helps in effectively identifying defects and ensuring thorough coverage.

💡Defect Density

Defect density measures the number of defects found per unit of code, such as per 1000 lines of code. It helps in evaluating the quality of the software and determining whether the exit criteria have been met. High defect density indicates areas that need more thorough testing.

💡Regression Testing

Regression testing involves re-testing existing test cases to ensure that new changes have not introduced any new defects. This approach is essential for maintaining the stability and reliability of the software, particularly after updates or modifications.

Highlights

The test planning process includes 10 key activities, starting with identifying the scope, risks, and objectives of testing.

Choosing a test planning approach can be either proactive or reactive.

Integration and coordination of test activities within the software lifecycle are crucial, often following the V-model.

Deciding which parts of the system to test is a critical step in the planning phase.

Scheduling is vital, covering task timelines, participants, and timeframes for implementation, execution, and evaluation.

Resources assignment and detailing the level of detail in tasks and metrics are essential for monitoring and controlling.

Monitor and control involve checking for deviations from the plan and taking corrective actions.

Entry criteria define when to start testing, ensuring readiness in terms of environment, tools, and test data.

Exit criteria determine when to stop testing, based on factors like coverage, risk, defect density, and project timelines.

Test estimation involves predicting effort and resources required, using matrix-based and expert-based approaches.

Product characteristics, development process, and testing outcomes significantly impact the testing effort.

The choice of test strategy and approach, whether proactive or reactive, depends on context and available resources.

There are seven typical test approaches, including analytical, statistical, methodical, and consultative methods.

Analytical approaches focus on risk-based testing, targeting high-risk modules.

Regression-averse approaches emphasize reusing existing test materials and conducting regression testing.

Transcripts

We've talked before about the test management

and had a quick overview of the test planning and what does it mean.

let's know now the test planning activities.

They are 10 activities:

The first step is to identify the scope of testing in which you'll work

and what are the risks you'll face, and what are

the objectives of testing? And to mention

which test planning approach we will use, either

proactive or reactive. The third thing we do

is to integrate and coordinate the test activities inside

the software life cycle. Since, we've mentioned before, each life cycle activity

has in the opposite its testing activity in the V-model.

In this stage, we will put all the testing activities we have

in the cycle of the software development

Then Comes step no. 4

that is to decide which parts we need to test.

All this time , we were still talking about the planning stage to know

what to test, which roles we will need

which will execute all the testing activities

and how to evaluate the test results .

We start by setting a schedule. In planning,

we should always schedule for each task;

when it will happen, who will participate,

the time it takes, when it will start and end.

you should also schedule

the implementation, the execution and the evaluation.

when you are still in planning, don't forget to assign all the resources

of each activity and identify the level of details

you'll work on and select which metrics

you will need during the monitor and control.

The "monitor and control" is the follow-up

we do to monitor and to check if what

you execute in reality is the same as what .

you planned for while the control means taking an immediate action,

to fix any defect and to solve

any problem at once, whenever you find a deviation

between plan and execution. what do we do

in control? It helps us to take an action to improve

ourselves during working on the test process.

You should mention the level of details in the test procedures

in order to tell those who will do the task of preparation

and execution where to start in the test planning.

You must also identify the entry criteria

on which you will start testing, in addition to

identifying the exit criteria. The entry criteria

means "when to start testing?"

for example :I will start testing at the beginning

of each test level.

or when I have a test group which is about execution.

what do the Entry criteria cover?

when you start testing, what will you need?

To start testing, can you start without

an environment which is ready for testing?

or can you start testing while they are not ready?

So I want the test environment to be at least available,

any tools or the test data we need,

or anything we need as a setup to the environment.

Whether what we are going to test ready or not?

If we are using tools, are these tools ready for

the test environment or not?

Testable code: If we work the white box, is our code

and the test data are available or not?

All tools we need before starting testing process,

are called the Entry Criteria. You must ensure

that they are available so as not to be blamed later on when people say

"you wasted time". If you started testing

without test preparation for your data,

you will search for test data.

Also if you start testing while you don't have the tool you will use.

All that will affect the timing of testing

and this is what we call the Entry Criteria.

the stage before testing is like preparing a checklist for

the things I'll need in testing, are they available or not?

If anything is not available, you will stop and ask them to prepare it.

when will we finish testing?

we won't test forever.

you must determine when to start testing

and when to stop testing. I will stop the testing process

at the end of each test level,

or when we finish each group of test on a specific goal.

For example,if we finish registration today

so this will be the end or the point you will stop at

when we reach the ending of registration.

The Exit criteria: where should you focus?

I will focus on "Did we cover our code and functionality

or the risk?" what does it mean? I said here I will stop

testing when I reach 100*100 decision coverage

I am the one who set that. So I'll get back to

the question "did I reach this objective or not?"

I will stop testing when I reach 90%

of the test cases have been executed,

and this is for me measure.

Or I'll stop testing based on the risk analysis

as the most dangerous 2 modules are 1&2

and we've finished testing them. So this is for me

my Exist criteria. This is regarding the 1st point.

Your exist criteria can be defect density.

what is defect density? it's the number of defects we found

per end line of codes.Again,

we test say 1000 lines of code if we found

500 defects per 1000 lines of code

this is for me Exist criteria and I will stop testing here.

The Exist criteria can be

number of the remaining risks, like what?

i.e.what can remain from them?

Do we fix all defects we find at once?

No, I may prioritize the defects I find

and may leave some of them to be done later.

So my Exist criteria may be the defects not fixed

or lack of test coverage in certain areas.

Some areas in the application I didn't give them full coverage.

You may need to finish testing

because of the time to market.

I need to publish the first version so I'll stop testing.

So what factors that determine the Enter and Exist criteria

The availability:

Is the team ready? Are the environment and system ready?

Are the tools available?

and the test items status to start or to end testing

Defects,

measurements, defect trend and severity.

What is it meant by defect trend ?

As long as the project is ongoing, we check if the defects we found are relative to the defect fixed.

if the defect is opened i.e. found, so how much did we finish?

Test execution measurement

what is the measurement of the test execution?

How many passed and failed tests? and how many tests we run?

Coverage measurements

how much did you cover from the requirements and the code?

All that is in terms of coverage measurements.

Quality characteristics measurement

what are the quality attributes which you have checked?

the finding defects cost and our budget

is the exist criteria for me.

what is the risks of delivery of the early or

the late modules or software?

Now, we've finished all about planning

what happen in it and its activities

and Entry and Exist criteria. Now, we will move to a stage

called: Test Estimation. when a test leader

works with say a team of five testers,

he is required to estimate the effort

submitted from each tester in terms of

number of the tasks which assigned to him

and the timing set to each task. The estimation is not

haphazard, i.e. it is not his evaluation. they work according to

some approaches. These approaches can help him

to estimate tasks so as not to be unfair with his team members

by giving one some tasks above

his capabilities, or setting timing less than the task needs.

There are many approaches for the estimation .

we have only 2 of them:

Matrix based and expert based

he shall use the Matrix for estimating

the testing effort on the similar projects.

and check what time did it take and according

to this basis we start estimating. This is one of the ways.

There is another way , which is to estimate the test effort

based on the estimation made by the task owner

or by an expert.

He sets the time for a task to be 2 or 3 days.

So on what it is based here ?!

It is based on the experience of the owner of

this task. Once the test effort was estimated,

we will start make assignments to

the resources it will need.

Then we start to set a schedule for him to follow.

The testing effort depends on a number of factors.

includes: Characteristics of the product

and the characteristics of the development process

and the outcome of testing. So there are some elements

affect the test effort during working.

What are the characteristics of the product?

The quality of specification and information resulted from

the test basis we have,

the size of the product, the complexity

of the problem domain... all those elements

affect the effort. Which principle of testing

does it match, if you remember?

It matches contacts dependence.

According to the contacts, the way should be different.

and as the way is different, so the effort will be different.

Also the resources, we will work on or we may need,

will be different.

One of the factors that may affect the test effort

is the development process and stability

of the organization.

Does the effort I need in working for a professional

environment gives me enough time and resources for testing?

Or they conduct this testing unprofessionally?

Or they ask me to finish quickly under pressure,

while they don't have any processes.

Of course the development process also affects

the testing effort, as we don't work in isolation.

Testing is strongly affected with the development process.

The third element affects the test effort is the outcome of testing

which is the number of defects we found

and the amount of re-work required.

So, Whenever we find a problem that affects the product,

we are obliged to get it back to be fixed. Then it come back to be reviewed again.

All these factors affect the testing effort.

Here, We talk again about the test strategy and the test approach

This is what we call test strategy,

and the implementation of test strategy for specific project.

where is this test approach defined?

It is defined in the test plan and the test design.

in which we mention whether we work proactive or reactive.

In the starting point of a test process,

we should select the test design techniques

and the test types in which we mention and identify

as our entry and exist criteria.

On which basis do you select the approach? As you like?

No, It's based on the context. As we previously

mentioned that test is context dependent.

If there is a risk or safety critical

and based on the available resources,

will we start working early or at the very last end ?

The skills of other participants, the technologies, and the

type of system ; do we work COTS?

what are the testing objectives?

are there any regulations on you or not?

There are typical approaches.

There are 7 approaches, about which you are always asked.

what is required from these seven is to know

the differences between them. The first one is

the analytical approach. This one depends on my analysis

for the risk of the system. This is what we call

the risk based testing.

The test effort is mostly for

the modules that have the greatest risk.

So the analytical one analyzes system and

identifies the high risk modules, then it starts to direct the testing effort

towards them. The second approach is

to work on some statistical models or use

"The stochastic testing"

what is the stochastic testing? Here we use some

statistical data about the failure rate, on which

we will work. the stochastic testing has another

name which is "random testing".

It means that we input multiple inputs in a definite sequence

to the system, then check the system response.

In other words, by using this approach,

It will measure the reliability rate of the system

comparing to the bugs we find and fix in system.

There is another term called "operational profiles"

What is the operational profiles?

It is a model acts like a guide for

the software development in order to

help me to know which parts of

the software are operational and

non-operational.i.e. how much have we finished

of the development and what remains?

The third type is called

the methodical approaches.

This one deals with failures.

It includes an error guessing and fault attacks

This approach depends on the experience of the test engineer

This is an experience based technique, on which we determine

the way of testing or the approach of testing.

We may find another approach

believes that the way of testing is related

to the process or the standard complaince.

Do we follow this process or not?

like the industry specific standards

i.e. if we test medical context, we will follow our testing.

So we will be directed to leading

the available industry standards.

The Dynamic and heuristic

approaches, this is the fifth type.

Here, we work exploratory

which relies on specific hypotheses

based on my experience.

Some people use in testing an approach

called "consultative approaches".

This means that I get some

experts in business domain from outside the team.

Then I get their consultation test

on the coverage which we've finished.

Based on the experience of some people from domain

but out-side the test team.

Here we have a different approach called

regression averse approaches.

It is to reuse or retest all the existing test material

while you do the automation

which was modified.

then I will conduct regression testing and standard test all the time.