ISTQB CTAL TAE Session 36 - 5.1 - Selection of TAS Metrics

Summary

This training series episode delves into the pivotal aspect of test automation reporting and metrics. It introduces the importance of monitoring indicators for effective test automation, covering the selection of relevant metrics, their implementation, and the significance of comprehensive logging. The episode also distinguishes between external and internal test automation metrics, emphasizing the benefits of automation and how they can be measured, such as time savings, increased coverage, and early defect detection, ultimately highlighting the value added to the testing process.

Takeaways

• 🚦 The video discusses the importance of test automation reporting and metrics, comparing it to a car dashboard that provides essential information for driving.
• 📊 The script is divided into four key sections: selection of T metrics, implementation of measurement, logging of T and Su, and test automation reporting.
• 🔍 Section 5.1 focuses on choosing the most relevant metrics for a test automation project to monitor the strategy and effectiveness of the automation.
• 📈 T metrics are divided into external and internal metrics, with external metrics measuring the impact on testing activities and internal metrics assessing the automation's performance.
• 🛠️ External T metrics include automation benefits, effort to build and maintain tests, test incident analysis, and code coverage, among others.
• 🔧 Internal T metrics involve tool scripting metrics, automation code defect density, and the speed and efficiency of T components.
• 💡 The benefits of test automation should be clearly communicated, as they might not be immediately visible, especially to those not directly involved in the process.
• 📉 Possible measures of automation benefits include efficiency gains, streamlined regression testing, expanded testing scope, and early defect detection.
• 🔄 The script emphasizes that the benefits of test automation align with the original goals set for the automation project, such as saving time, enhancing coverage, or reducing errors.
• 📝 It's important to distinguish between T metrics, which are managed by the test automation manager and engineer, and Su related metrics, which are managed by the project's overall test manager.
• 🌐 Measuring the benefits of test automation reveals its true value, including efficiency, accuracy, and agility, and encourages further leveraging of its potential.

Q & A

What is the main focus of the chapter on test automation reporting and metrics?

The chapter focuses on the information and numbers gathered from automated tests, similar to a car dashboard, to monitor various indicators and ensure everything is on track in test automation.

How many key sections are there in the exploration of test automation reporting and metrics?

There are four key sections: selection of T metrics, implementation of measurement, logging of the T and Su, and test automation reporting.

What does Section 5.1 focus on in the context of test automation?

Section 5.1 focuses on the selection of the most relevant metrics for a test automation project, specifically for monitoring the test automation strategy and its effectiveness and efficiency.

What are the two main things that test automation strategy metrics are used to assess and keep track of?

Test automation strategy metrics are used to assess and keep track of how well the planned approach for automating tests is working and the effectiveness and efficiency of the chosen automation strategy in achieving its intended goals and objectives.

What is the distinction between test automation metrics and Su related metrics?

Test automation metrics are used to track the progress of the test automation effort and evaluate its performance, managed by the Test Automation Manager (TAM) and Test Automation Engineer (TAE). Su related metrics, on the other hand, monitor the system under test and its functional and non-functional aspects, managed by the Project's overall test manager.

What are the two categories of test automation metrics mentioned in the script?

The two categories of test automation metrics are external T metrics, which measure the impact of T on other activities, particularly testing activities, and internal T metrics, which measure the effectiveness and efficiency of the T in fulfilling its objectives.

What is the significance of measuring automation benefits in test automation?

Measuring automation benefits is important to highlight the value and advantages of test automation, such as time savings, increased testing coverage, and reduced errors, which may not be immediately visible to everyone, especially those not directly involved in the process.

How can the benefits of test automation be measured?

Benefits can be measured through efficiency gains, number of hours of manual test effort saved, streamlined regression testing, enhanced test execution, expanded testing scope, and early defect detection, among others.

What is the purpose of calculating the test automation ratio?

The test automation ratio, which is the percentage of automated test cases related to the entire set of test cases, helps to understand the balance between automated and manual testing in the process, although it does not indicate which is better or more effective.

How does test automation enable additional manual testing?

Test automation saves manual test effort, which can be devoted to other kinds of manual testing, such as exploratory testing. Defects found by these additional tests can be seen as indirect benefits of test automation.

What is the conclusion of the session on test automation reporting and metrics?

The conclusion emphasizes that measuring the benefits of test automation reveals its true value, acknowledging the efficiency, accuracy, and agility it brings to the testing process, and inspiring continued leveraging of its potential for more impressive outcomes.

Outlines

00:00

📊 Test Automation Reporting and Metrics

This paragraph introduces the concept of test automation reporting and metrics, emphasizing their importance in the test automation journey. It likens test automation metrics to a car's dashboard, providing essential information about the performance and health of the automated tests. The chapter will cover four key sections: selection of test automation metrics, implementation of measurement, logging of test and system under test (SUT) metrics, and test automation reporting. The focus is on choosing relevant metrics to monitor the test automation strategy's effectiveness and efficiency, as well as the separate metrics for the SUT. The paragraph also distinguishes between test automation metrics, managed by the test automation manager and engineer, and SUT-related metrics, overseen by the project's test manager.

05:02

🔍 Understanding External and Internal Test Automation Metrics

This paragraph delves into the specifics of external and internal test automation metrics. External metrics measure the impact of test automation on broader testing activities, while internal metrics assess the effectiveness and efficiency of the test suite in achieving its objectives. The paragraph lists and briefly explains various external metrics such as automation benefits, effort to build and maintain automated tests, and code coverage. It also mentions internal metrics like tool scripting metrics, automation code defect density, and the speed and efficiency of test components. The importance of making the benefits of test automation clear to stakeholders who may not see the immediate advantages is highlighted, with examples of how to measure these benefits, including efficiency gains, streamlined regression testing, and expanded testing scope.

10:03

🛠️ Measuring the Benefits and Impact of Test Automation

The final paragraph discusses the importance of measuring the benefits of test automation, such as increased coverage, early defect detection, and unique defect discovery that might not have been found through manual testing. It explains how these benefits can be quantified, for example, by calculating cost savings from finding defects earlier in the development process or by assessing the number of additional tests that can be executed due to time saved by automation. The paragraph also touches on the indirect benefits of test automation, like enabling more comprehensive testing and deeper exploration, which can uncover issues that might have remained hidden. The session concludes by emphasizing the value of test automation in improving efficiency, accuracy, and agility in the testing process, and it sets the stage for further exploration of test automation metrics in the next session.

Keywords

💡Test Automation

Test automation refers to the process of using specialized software to execute tests on a system or application. In the script, test automation is the central theme, emphasizing its importance in modern software testing practices. It is discussed in the context of its strategy, effectiveness, and the metrics used to measure its success.

💡Reporting and Metrics

Reporting and metrics are essential in test automation to provide insights into the performance and effectiveness of automated tests. The script describes these as vital components that help in making informed decisions and tracking progress, similar to a car's dashboard that provides real-time information about the vehicle's status.

💡T Metrics

T metrics, as mentioned in the script, are specific metrics used to assess the performance of test automation. They are divided into external and internal metrics, which measure the impact of test automation on testing activities and the effectiveness and efficiency of the test suite itself, respectively.

💡External Metrics

External metrics are used to measure the broader impact of test automation on testing activities. The script lists several external metrics such as automation benefits, effort to build automated tests, and code coverage. These metrics help in understanding how test automation influences the overall testing process.

💡Internal Metrics

Internal metrics focus on the efficiency and effectiveness of the test suite in achieving its objectives. The script mentions tool scripting metrics, automation code defect density, and the speed and efficiency of test components as examples of internal metrics, which are crucial for assessing the performance of the test automation solution.

💡Automation Benefits

Automation benefits are the advantages gained from implementing test automation. The script highlights the importance of quantifying these benefits, such as time savings, increased test coverage, and reduced manual errors, to demonstrate the value of test automation to stakeholders.

💡Efficiency

Efficiency in the context of the script refers to the effectiveness and productivity of the test automation process. It is used to evaluate how well the chosen automation strategy is achieving its intended goals, such as reducing the time and effort required for testing.

💡Test Incidents

Test incidents are events or occurrences during the testing process that may require attention or action. The script discusses the metric of 'effort to analyze automated test test incidents,' indicating the importance of understanding and addressing issues that arise during automated testing.

💡Maintenance

Maintenance in the script refers to the ongoing effort required to keep automated tests up-to-date and effective. It is a key aspect of test automation, as it ensures that tests remain relevant and accurate over time, which is measured by the 'effort to maintain automated tests' metric.

💡Code Coverage

Code coverage is a metric that measures how much of the software's code is executed during testing. The script mentions it as one of the external metrics, indicating its importance in assessing the thoroughness of test automation and ensuring that a wide range of code is tested.

💡Actionable Insights

Actionable insights are the practical and useful conclusions that can be drawn from the data collected through test automation reporting and metrics. The script emphasizes the creation of informative reports that convert data into insights that can guide the test automation efforts effectively.

Highlights

Introduction to the new chapter on test automation reporting and metrics.

Comparison of test automation metrics to a car dashboard for monitoring progress.

Exploration of four key sections on test automation reporting and metrics.

Section 5.1 focuses on the selection of test automation metrics.

Importance of choosing relevant metrics for a test automation project.

Section 5.2 discusses the implementation of measurement in test automation.

Section 5.3 emphasizes the importance of comprehensive logging in test automation.

Section 5.4 is about creating informative test automation reports.

Differentiation between test automation metrics and system under test (SU) related metrics.

Responsibility for managing SU related metrics lies with the project's overall test manager.

Test automation metrics track the progress of test automation efforts.

External and internal test automation metrics explained.

External metrics measure the impact of test automation on testing activities.

Internal metrics assess the effectiveness and efficiency of the test suite.

List of external test automation metrics provided, including automation benefits and effort to build automated tests.

List of internal test automation metrics, such as tool scripting metrics and automation code defect density.

Explanation of automation benefits and its importance in reporting the value of test automation.

Benefits of test automation depend on the objectives set for the test suite.

Possible measures of automation benefits include efficiency gains and hours saved by automating tests.

Discussion on the balance between automated and manual testing methods.

Coverage amplification and early defect detection as benefits of test automation.

Unique defect discovery as a significant advantage of test automation.

Indirect benefits of test automation through enabling additional manual testing.

Conclusion on the importance of measuring the benefits of test automation for a comprehensive understanding of its value.

Transcripts

welcome back to Tae certification

training Series today we are about to

start a new chapter in our test

automation Journey so far we have

covered a lot from why we automate test

to getting ready for its execution and

exploring the complexities of a journal

test automation framework now we are

about to dive in something equally

important but different which is test

automation reporting and

metrics in plain terms this chapter is

all about the information and numbers we

can gather from our automated test it is

like a dashboard in your car when you

drive you need to know how fast you are

going how much fuel you have and whether

the engine is working well or not

similarly in test automation we need to

keep an eye on various indicators to

ensure that everything is on track we

will explore test automation reporting

and metrics through four key

sections the first one is 5.1 selection

of T metrics this section focuses on

choosing the most relevant metrics for

your test automation

project then we have 5.2 implementation

of measurement in this section you will

learn how to put measurement into action

and gather valuable data

effectively next is 5.3 logging of the T

and and the Su in this section you will

discover the importance of comprehensive

logging in both the T and the

Su and finally 5.4 test automation

reporting in this section you will

explore how to create informative

reports that turn data into actionable

insights for your automation

efforts so let's start with Section 5.1

selection of Tes metrics and discover

how to choose the write metrics to guide

your test automation

Journey this section focuses on metrics

for monitoring the test automation

strategy and the Effectiveness and

efficiency of the

T so it's primarily concerned with

discussing metrics used to assess and

keep track of two main things mentioned

over here the first one is test

automation strategy which is about

monitoring how well the planned approach

for automating test is working and the

second is Effectiveness and efficiency

of the T it involves evaluating how

effective and efficient The Chosen

automation strategy is in achieving its

intended goals and

objectives next we have separate from Su

related metrics used to monitor the

system under test and the functional and

non-functional testing of the Su which

is selected by the Project's overall

test manager so the key Point here is

that these automation metrics are

distinct and separate from another set

of metrics known as Su related metrics

these Su related metrics are used to

monitor this Su which is system under

test which is the software or

application which is being tested along

with its functional and non-functional

aspects and the responsibility for

selecting and managing these Su related

metrics lies with the Project's overall

test

manager test automation metric allow the

Tam and Tae to track progress towards

the goals for test Automation and to

monitor the impact of changes made to

the test automation

Solution on the other hand the test

automation metrics are the domain of the

Tam test automation manager and the Tae

test automation engineer these metrics

are used to track the progress of the

test automation effort and to evaluate

how changes made to the test automation

solution are impacting its performance

so to summarize test automation metrics

are like the instruments for gauging the

efficiency of the automation strategy

while Su related metrics focus on the

quality and performance of the software

being tested each set of metrics serves

a distinct purpose and is managed by

different roles within the testing

team next we have t metrics and there

are two of them external and

internal so the external T metrics are

used to measure the T impact on other

activities in particular the testing

activities and the internal one used to

measure the Effectiveness and efficiency

of the TS in fulfilling its

objectives think of it this way external

metrics show how your Tas affects the

bigger picture of testing while internal

metrics tell you how well the TS is

doing its job job it's like looking at

both the forest and the trees to

understand the whole

picture so let's have a look at all the

external Tes metrics over here and then

later on we will go through all of them

one by one in detail the first one is

automation

benefits number two effort to build

automated

test three effort to analyze automated

test test

incidents four effort to maintain

automated

tests five ratio of failures to

defects six time to execute automated

tests seven number of automated test

cases eight number of pass and fail

results nine number of false fail and

false pass

results and number 10 code coverage

and similarly we have a list of internal

TS metrics too the first one is tool

scripting

metrics number two automation code

defect density and the third one speed

and efficiency of T

components let's now have a look uh at

all the external T metrics one by one

and the first one is automation benefits

why mayor automation benefits it is

particularly important an to Mayor and

Report the benefits of a

t this is because the costs in terms of

the number of people involved over a

given period of time are easy to

see people working outside testing will

be able to form an impression of the

overall cost but may not see the

benefits

achieved think of your TS as a toolkit

you have carefully assembled you have

put in time and effort to make it work

seamlessly now it's time to reveal the

real trayers hidden within this toolkit

which is the benefits of test automation

now here's the deal while it's quite

clear how much effort and resources go

into setting up testing automation like

the time and people involved the

benefits might not be immediately

visible to everyone especially uh those

who are not directly involved in the

process that's why we need to shine a

light on these benefits making them

crystal clear for everyone to

see benefits depend on T

objectives any mayor of benefit will

depend on the objective of the

T typically this may be a savings of

time or effort an increase in the amount

of testing performed breadth or depth of

coverage or frequency of execution or

some other Advantage such as increased

repeatability greater use of resources

or fewer manual

errors so the reward you reap from your

TS depend on what you aimed to achieve

in the first place did you set out to

save time enhance testing coverage or

reduce errors the benefits align with

your original

goals possible measures include

efficiency gains number of hours of

manual test effort saved so count the

hours saved by automating test compared

to manual testing

efforts streamlined regression testing

reduction in time to perform regression

testing so highlight how much faster

regression testing becomes with

automation enhanced test execution

number of additional cycles of test

execution achieved track how many

additional testing Cycles you can

execute thanks to

automation expanded testing scope number

or percentage of additional test

executed measure the extra test you can

perform due to

automation test automation ratio

percentage of automated test cases

related to the entire set of test cases

although automated cannot easily be

compared to manual test

cases okay imagine you have a group of

test cases some of these test cases are

automated While others are manual now

you want to know what portion of your

total testing effort is done through

automated test cases from a pool that

includes both automated and manual test

however here is the catch automated and

manual test cases are like apples and

oranges they serve different purposes

and are used in different situations so

directly comparing them is not

straightforward so while you can

calculate this percentage to understand

the balance between automated and manual

testing in your process remember that it

doesn't tell you which is better or more

effective it's a simple way to see how

you are using these two types of testing

methods then we have coverage

amplification increase in coverage

requirements functionality

structural so assess the broader

coverage you achieve whether it's

related to requirements functionality or

structural

aspects then we have early defect

detection number of defects found

earlier because of the T when the

average benefit of defects found earlier

is known this can be calculated to a sum

of prevented costs so calculate the cost

savings by finding defects earlier in

the development process thanks again to

automation then we have unique defect

Discovery number of of defects found

because of the T which would not have

been found by manual testing for example

reliability defects in this one identify

defects uh uncovered solely because of

automation which might have been missed

in manual

testing additional consideration note

that test automation generally saves

manual test effort this effort can be

devoted to other other kinds of manual

testing for example exploratory

testing defects found by these

additional test can also be seen as

indirect benefits of the T as the test

automation enabled these manual tests to

be executed without the T these test

would not have been executed and

subsequently the additional defects

would not have been

found here is the exciting twist test

automation not only saves time and

effort but also opens up opportunities

for more comprehensive testing and

deeper exploration it's like having a

versatile toolkit that allows you to

explore different testing scenarios and

uncover issues that might have otherwise

stayed hidden in the end measuring the

benefits of test automation is like

revealing the true value of your testing

efforts it's about acknowledging the

efficiency accuracy and Agility it

brings to your testing process and

inspire you to continue leveraging its

potential for even more impressive

outcomes this concludes today's session

we will continue exploring the remaining

TS metrics in our next session until

then take care