CH05. L04. Test monitoring and control

Summary

This script delves into the crucial aspects of test progress monitoring and control within a project. It outlines the roles and responsibilities of a test team and leader, emphasizing the importance of test planning and its implementation. The video discusses various monitoring methods, from manual to automated tools, and the significance of test metrics to gauge progress. It also covers test control actions, including reactive and preventive measures, and concludes with the creation of a test summary report, adhering to IEEE829 standards, to communicate comprehensive findings and recommendations to stakeholders.

Takeaways

• 📈 **Test Progress Monitoring**: The test manager observes and compares the actual progress of test activities against the planned results.
• 🔍 **Monitoring Methods**: The choice between manual and automated tools for monitoring depends on the team size and project scale.
• 📊 **Test Metrics**: Eight key metrics are used for monitoring, including test case preparation, execution, defect density, and test coverage.
• 🔎 **Defect Information**: Monitoring helps in understanding the number of defects found, fixed, and the failure rate, which is crucial for quality assessment.
• 📝 **Test Environment**: The preparation of the test environment and its impact on the overall testing process is a monitored aspect.
• 📉 **Defect Density**: A metric that measures the number of bugs found relative to the lines of code, indicating software quality.
• 📚 **Test Coverage**: The extent to which test cases cover requirements, risks, or code is a critical metric for comprehensive testing.
• 🤔 **Subjective Confidence**: The level of confidence a tester has in the testing process, which can be an indicator of test quality.
• 🗓 **Test Milestones**: Monitoring includes tracking the dates of test milestones to ensure the project stays on schedule.
• 💰 **Test Costs**: Understanding the costs associated with testing can help in decision-making regarding further defect searching or test execution.
• 🛠 **Test Control**: After monitoring, identifying deviations from the test plan and implementing corrective or preventive measures is essential.
• 📋 **Test Summary Report**: A comprehensive report submitted to stakeholders, analyzing results, metrics, and providing recommendations for future actions.
• 📘 **IEEE829 Standard**: The test summary report follows the IEEE829 standard, ensuring a structured and standardized format.
• 📝 **Report Components**: The report is divided into eight parts, including identification, summary, variations, assessment, evaluation, activities summary, and approvals.

Q & A

What is the primary role of a test manager in the context of test planning?

The test manager is responsible for setting the test planning to be followed during the project and overseeing the activities related to it.

Why is monitoring test progress important?

Monitoring test progress is crucial for providing feedback on test activities, observing the progress of test planning activities, and comparing it to the actual plan results.

How does the choice of monitoring method depend on the team size?

For smaller teams, manual monitoring through documents, spreadsheets, or databases is feasible. However, larger teams working on larger projects require automated tools for efficient data gathering and monitoring.

What are the eight important points that test metrics measure during monitoring?

Test metrics measure the percentage of work done in test case preparation and execution, test environment preparation, defect information, defects density, defects found and fixed, failure rate and retest results, test coverage of requirements, risks, or code, subjective confidence level of testers, and test costs.

How does monitoring help in understanding the test cases' status?

Monitoring helps clarify the percentage of test cases that have been run, which ones have passed and failed, and provides an estimate of the work done in test environment preparation.

What is meant by defects density in the context of software testing?

Defects density refers to the number of found bugs in proportion to the number of lines of code in the software being tested.

What is the purpose of the test control stage after monitoring?

The test control stage involves identifying deviations from the test plan, deciding on recovery or preventive actions, and determining whether to correct issues or skip them based on their impact.

What are some common test control actions a manager can take?

Common test control actions include rescheduling testing, increasing the number of testers, re-ordering test priorities, and deciding which parts can be skipped without testing.

What is the significance of the test summary report?

The test summary report is important for submitting comprehensive reports to stakeholders, analyzing results and metrics, and setting recommendations for future actions.

What standard is commonly followed for the format of the test summary report?

The IEEE829 standard is commonly followed for the format of the test summary report.

Can you describe the eight parts of the test summary report as per IEEE829 standards?

The eight parts are: 1) Summary report identification, 2) Summary, 3) Variations, 4) Comprehensive assessment, 5) Overall status of incidents, 6) Evaluation, 7) Summary of activities, and 8) Approvals.

Outlines

00:00

📊 Test Progress Monitoring and Control

This paragraph discusses the importance of monitoring and controlling test progress within a project. It highlights the roles and responsibilities of a test manager in setting and following test plans, as well as gathering feedback on test activities. The paragraph emphasizes the need for a monitoring method chosen by the team, which could be manual for small teams or automated for larger projects. It also outlines the use of test metrics to measure eight key points, including the percentage of test cases prepared and executed, defect density, and test coverage. The paragraph concludes with the discussion on test control actions, such as rescheduling and resource allocation, and the significance of a test summary report in providing comprehensive insights to stakeholders and setting recommendations for future actions.

05:03

📝 Test Summary Report Structure

The second paragraph delves into the structure and purpose of a test summary report, as per IEEE829 standards. It underscores the test manager's responsibility in creating this report during the project's execution phase. The summary report serves to submit comprehensive project test process reports to stakeholders and to analyze results and metrics for future recommendations. The paragraph outlines the eight-part structure of the report, starting with identification and including sections on what was tested and its results, deviations from the plan, comprehensive assessment of test objectives, overall incident status, evaluation of quality based on test plan criteria, summary of activities, and finally, the approvals section with signatures and authority confirmations.

Keywords

💡Test Progress Monitoring

Test Progress Monitoring is the process of tracking the status and progress of testing activities against a predefined plan. It is crucial for ensuring that testing is on schedule and meeting its objectives. In the script, it is mentioned as a necessary step after setting up test planning, where the test manager observes and compares the actual progress with the planned results, using either manual or automated tools depending on the team size and project scale.

💡Test Manager

A Test Manager is a professional responsible for overseeing the testing process, including planning, execution, and reporting. They are tasked with setting the test plan and ensuring that all testing activities are carried out as planned. In the script, the Test Manager's role is highlighted in monitoring test progress, choosing the appropriate monitoring method, and taking corrective actions when needed.

💡Test Planning

Test Planning is the phase where the strategy and schedule for testing activities are defined. It includes determining the scope, resources, and approach for testing. The script refers to test planning as the groundwork laid by the Test Manager, which is then followed and monitored throughout the project.

💡Feedback

In the context of the script, Feedback refers to the information gathered during the monitoring stage that provides insights into the performance and effectiveness of the testing activities. It helps the Test Manager understand how the actual test activities compare to the planned ones and make informed decisions for improvement.

💡Test Metrics

Test Metrics are quantitative measures used to assess the quality of testing and the progress of test activities. The script mentions eight important points that test metrics can measure, such as the percentage of test cases prepared and executed, defects density, and test coverage, which are vital for evaluating the testing process.

💡Defect Density

Defect Density is a metric that indicates the number of defects found in relation to the size of the software, typically measured against the number of lines of code. It helps in understanding the quality and reliability of the software under test. The script uses this term to illustrate one of the key metrics that can be monitored during testing.

💡Test Coverage

Test Coverage refers to the extent to which the software has been tested against its requirements, risks, or code. It is a critical metric that helps determine how comprehensive the testing has been. In the script, test coverage is mentioned as one of the important points measured by test metrics.

💡Test Control

Test Control involves taking corrective actions to address deviations from the test plan and to prevent issues in future stages. It is about being 'reactive' to current problems or 'preventive' to avoid future ones. The script discusses test control as a phase following test monitoring, where the Test Manager identifies deviations and decides on recovery strategies.

💡Test Summary Report

A Test Summary Report is a comprehensive document that summarizes the testing process, results, and metrics. It serves to communicate the findings to stakeholders and provides recommendations for future actions. The script emphasizes the importance of this report in the final stage of the test process, outlining its objectives and the standard format based on IEEE829.

💡IEEE829

IEEE829 is a standard that provides a format for test documentation, including the Test Summary Report. It ensures consistency and completeness in reporting. The script mentions IEEE829 as the standard followed for structuring the Test Summary Report, highlighting its eight-part format.

💡Stakeholders

Stakeholders are individuals or groups with an interest or concern in the project's outcome. In the context of the script, stakeholders are the recipients of the Test Summary Report, who rely on this documentation to understand the testing process and its results. The report's objectives include submitting comprehensive reports to stakeholders.

Highlights

Test progress monitoring and control are essential for relating all issues together in the testing process.

Understanding the rules and responsibilities of a test team, including the test leader's role in setting and following test plans, is crucial.

Monitoring test progress involves comparing actual results with the test plan to ensure alignment.

The choice of monitoring method depends on the team size and project scale, ranging from manual to automated tools.

Test metrics measure eight key aspects, including test case preparation, execution, and defect information.

Monitoring helps clarify the percentage of test cases run, passed, and failed, and defect density.

Defects found, fixed, and the failure rate are essential metrics for understanding test progress.

Test coverage of requirements, risks, or code is a critical metric for assessing testing thoroughness.

Subjective confidence levels of testers provide an indication of the perceived quality of testing.

Matrix monitoring provides dates of test milestones, aiding in planning and decision-making.

Test costs are an important metric to determine the continuation of defect searching and testing.

The monitor stage is divided into quantity and calculation, covering test case numbers, defect numbers, and subjective confidence levels.

Calculations such as failure rate, defects density, and defect detection percentage are vital for analyzing test effectiveness.

Test control involves observing deviations from the test plan and implementing corrective or preventive actions.

Common test control actions include re-scheduling, increasing tester numbers, and prioritizing test cases.

The test summary report is a comprehensive document detailing the test process, results, and metrics.

The summary report serves to inform stakeholders, analyze results, and provide recommendations for future actions.

The IEEE829 standard outlines the format for the test summary report, consisting of eight parts.

The report includes identification, summary, variations, comprehensive assessment, overall status of incidents, evaluation, summary of activities, and approvals.

Transcripts

We'll talk about Test progress monitoring and control

In order to relate all issues together,

let us mention quickly that

we've learned the rules and responsibilities

of a test team,

the responsibility of a test leader

or a test manager towards the task,

as he set the test planning

to be followed during the project

as well as the activities.

Now, when we start implementing the rest of a test planning

we will need to monitor

the test progress and get a feedback

about the test activities of a tester.

So in the test monitor stage,

the test manager observes the test planning

activities progress,

then he compares it to the actual plan results.

Our choice of a monitoring method

is set by the team.

If a project team has few members,

I will be able to monitor manually,

either by documents, spreadsheets, databases.

However, if we work in a large team for a large project,

in this case we will need a tool

which is on a high a level of efficiency

in order to gather all the monitoring data

so it needs to be done by

automated tools. The test metrics

can be also used during monitoring

to measure 8 important points:

1-The percentage of both

the work done in test cases preparation

and the test cases execution.

Here, Monitoring is useful in

clarifying the percentage of

the test cases we've run

and the test cases which are

passed and failed.

2- We also can estimate the percentage of

work done in test environment preparation,

in addition to the defect information.

3-These information can help us

to know the defects density which means

that the number of found bugs

In proportion to the number of code's lines

of the software we testing

4- we can also know the defects found and which ones are fixed.

In addition, it helps us to know

the failure rate,

and the retest results.

5- It also helps us to know test coverage of requirements, risks

or code. 6- Moreover, it gives us

indication about subjective confidence

level of tester .

7- we'll find that matrix monitor

gives us the dates of test milestones.

8-Through this, We can know the test costs

that can assist me in deciding whether

to continue searching for the next defect

or when I'll be able to run

the next test.

Again, the matrix can divide

the monitor stage into

quantity and calculation.

This means that anything related to

the number of test cases which has been

run or not,

and the number of test cases

either passed or failed,

Also, the Number of defects which we've founded

and fixed, comparing the date of

the test milestones and the scheduled

with the reality, and finally

the subjective confidence level of testers.

these are all for the quality metrics.

what about calculations?

such as: failure rate,

defects density,

and defect detection percentage.

This was the test monitoring part.

So what about the test control?

After finishing the observation

and gathered the needed information

in the test monitoring stage.

So now, I need to find out the deviations that

happened in the test planning

and to know how to recover them

which is called "The Reactive ";

or at least, the test manager,

can prevent this problem from happening in the next stages.

This is called "preventive ",which depends

on the project itself.

Can we skip these bugs

or should we correct them?

Will these deviations affect the testing team?

So what are the common test

control actions that a manager can do?

He can re-schedule the testing.

or increase the number of testers.

and as a last solution, is to re-order the most important

to the least important.

In addition to which parts can be skipped without testing.

The test summary report:

Now we've finished what happens in the test planning

and its activities

and how to monitor and control

the test process.

The next stage is "summary report".

The test manager is the responsible for it

in the execution stage of the project.

So what is the benefit of the summary report?

we always make reports during the test process.

Note that:

the summary report has very important objectives.

- We use it to submit comprehensive reports to the stake holders

about the test process of the project.

- It also helps us to analyze results

and the metrics we gathered through which

we can set recommendations for future actions.

Does the summary report have a definite format to be followed?

yes, of course.

It is a common question in the ISTQB exam.

We always follow the IEEE829 standards

to submit the summary report.

So what is the format of our report?

It is divided into 8 parts:

First part is about the summary report

identification, in which we write the

name and the version of the project

The second part is the summary

in which we write the software tested version.

We also can add a list of tested items and

test environment and all data related to

what was tested and its results

whether interface or modules.

The third part is about variations.

what are the deviations that happened

out of the plan?

and what are their reasons?

The fourth part is comprehensive assessment.

Here we mention what was covered

and know what were my planned test

objectives

in the test planning.

and which objective were covered.

The fifth part:

in which we mention the overall status of

the incidents we found.

It's better in this part

not to use quantities, and use

calculations and ratios.

which should make it easier

to stakeholders.

The following part is about evaluation.

When we were setting the test plan

we had definite specifications

to be followed.

Here, we need to assess

the overall quality which resulted

based on the criteria specified

in the test plan.

Summary of activities: this part is related to

the stuff worked on the testing project.

whether on time or cost basis.

It compares what was written in

the test plan with what was actually done

during the run time.

Of course, if there is a difference between them,

we should mention the reasons of these variances .

The last part is the approvals which is

a list for all the signatures

and approval authority of the IEEE summary report.