ISTQB CTAL TAE Session 36 - 5.1 - Selection of TAS Metrics
Summary
TLDRThis training series episode delves into the pivotal aspect of test automation reporting and metrics. It introduces the importance of monitoring indicators for effective test automation, covering the selection of relevant metrics, their implementation, and the significance of comprehensive logging. The episode also distinguishes between external and internal test automation metrics, emphasizing the benefits of automation and how they can be measured, such as time savings, increased coverage, and early defect detection, ultimately highlighting the value added to the testing process.
Takeaways
- 🚦 The video discusses the importance of test automation reporting and metrics, comparing it to a car dashboard that provides essential information for driving.
- 📊 The script is divided into four key sections: selection of T metrics, implementation of measurement, logging of T and Su, and test automation reporting.
- 🔍 Section 5.1 focuses on choosing the most relevant metrics for a test automation project to monitor the strategy and effectiveness of the automation.
- 📈 T metrics are divided into external and internal metrics, with external metrics measuring the impact on testing activities and internal metrics assessing the automation's performance.
- 🛠️ External T metrics include automation benefits, effort to build and maintain tests, test incident analysis, and code coverage, among others.
- 🔧 Internal T metrics involve tool scripting metrics, automation code defect density, and the speed and efficiency of T components.
- 💡 The benefits of test automation should be clearly communicated, as they might not be immediately visible, especially to those not directly involved in the process.
- 📉 Possible measures of automation benefits include efficiency gains, streamlined regression testing, expanded testing scope, and early defect detection.
- 🔄 The script emphasizes that the benefits of test automation align with the original goals set for the automation project, such as saving time, enhancing coverage, or reducing errors.
- 📝 It's important to distinguish between T metrics, which are managed by the test automation manager and engineer, and Su related metrics, which are managed by the project's overall test manager.
- 🌐 Measuring the benefits of test automation reveals its true value, including efficiency, accuracy, and agility, and encourages further leveraging of its potential.
Q & A
What is the main focus of the chapter on test automation reporting and metrics?
-The chapter focuses on the information and numbers gathered from automated tests, similar to a car dashboard, to monitor various indicators and ensure everything is on track in test automation.
How many key sections are there in the exploration of test automation reporting and metrics?
-There are four key sections: selection of T metrics, implementation of measurement, logging of the T and Su, and test automation reporting.
What does Section 5.1 focus on in the context of test automation?
-Section 5.1 focuses on the selection of the most relevant metrics for a test automation project, specifically for monitoring the test automation strategy and its effectiveness and efficiency.
What are the two main things that test automation strategy metrics are used to assess and keep track of?
-Test automation strategy metrics are used to assess and keep track of how well the planned approach for automating tests is working and the effectiveness and efficiency of the chosen automation strategy in achieving its intended goals and objectives.
What is the distinction between test automation metrics and Su related metrics?
-Test automation metrics are used to track the progress of the test automation effort and evaluate its performance, managed by the Test Automation Manager (TAM) and Test Automation Engineer (TAE). Su related metrics, on the other hand, monitor the system under test and its functional and non-functional aspects, managed by the Project's overall test manager.
What are the two categories of test automation metrics mentioned in the script?
-The two categories of test automation metrics are external T metrics, which measure the impact of T on other activities, particularly testing activities, and internal T metrics, which measure the effectiveness and efficiency of the T in fulfilling its objectives.
What is the significance of measuring automation benefits in test automation?
-Measuring automation benefits is important to highlight the value and advantages of test automation, such as time savings, increased testing coverage, and reduced errors, which may not be immediately visible to everyone, especially those not directly involved in the process.
How can the benefits of test automation be measured?
-Benefits can be measured through efficiency gains, number of hours of manual test effort saved, streamlined regression testing, enhanced test execution, expanded testing scope, and early defect detection, among others.
What is the purpose of calculating the test automation ratio?
-The test automation ratio, which is the percentage of automated test cases related to the entire set of test cases, helps to understand the balance between automated and manual testing in the process, although it does not indicate which is better or more effective.
How does test automation enable additional manual testing?
-Test automation saves manual test effort, which can be devoted to other kinds of manual testing, such as exploratory testing. Defects found by these additional tests can be seen as indirect benefits of test automation.
What is the conclusion of the session on test automation reporting and metrics?
-The conclusion emphasizes that measuring the benefits of test automation reveals its true value, acknowledging the efficiency, accuracy, and agility it brings to the testing process, and inspiring continued leveraging of its potential for more impressive outcomes.
Outlines
📊 Test Automation Reporting and Metrics
This paragraph introduces the concept of test automation reporting and metrics, emphasizing their importance in the test automation journey. It likens test automation metrics to a car's dashboard, providing essential information about the performance and health of the automated tests. The chapter will cover four key sections: selection of test automation metrics, implementation of measurement, logging of test and system under test (SUT) metrics, and test automation reporting. The focus is on choosing relevant metrics to monitor the test automation strategy's effectiveness and efficiency, as well as the separate metrics for the SUT. The paragraph also distinguishes between test automation metrics, managed by the test automation manager and engineer, and SUT-related metrics, overseen by the project's test manager.
🔍 Understanding External and Internal Test Automation Metrics
This paragraph delves into the specifics of external and internal test automation metrics. External metrics measure the impact of test automation on broader testing activities, while internal metrics assess the effectiveness and efficiency of the test suite in achieving its objectives. The paragraph lists and briefly explains various external metrics such as automation benefits, effort to build and maintain automated tests, and code coverage. It also mentions internal metrics like tool scripting metrics, automation code defect density, and the speed and efficiency of test components. The importance of making the benefits of test automation clear to stakeholders who may not see the immediate advantages is highlighted, with examples of how to measure these benefits, including efficiency gains, streamlined regression testing, and expanded testing scope.
🛠️ Measuring the Benefits and Impact of Test Automation
The final paragraph discusses the importance of measuring the benefits of test automation, such as increased coverage, early defect detection, and unique defect discovery that might not have been found through manual testing. It explains how these benefits can be quantified, for example, by calculating cost savings from finding defects earlier in the development process or by assessing the number of additional tests that can be executed due to time saved by automation. The paragraph also touches on the indirect benefits of test automation, like enabling more comprehensive testing and deeper exploration, which can uncover issues that might have remained hidden. The session concludes by emphasizing the value of test automation in improving efficiency, accuracy, and agility in the testing process, and it sets the stage for further exploration of test automation metrics in the next session.
Mindmap
Keywords
💡Test Automation
💡Reporting and Metrics
💡T Metrics
💡External Metrics
💡Internal Metrics
💡Automation Benefits
💡Efficiency
💡Test Incidents
💡Maintenance
💡Code Coverage
💡Actionable Insights
Highlights
Introduction to the new chapter on test automation reporting and metrics.
Comparison of test automation metrics to a car dashboard for monitoring progress.
Exploration of four key sections on test automation reporting and metrics.
Section 5.1 focuses on the selection of test automation metrics.
Importance of choosing relevant metrics for a test automation project.
Section 5.2 discusses the implementation of measurement in test automation.
Section 5.3 emphasizes the importance of comprehensive logging in test automation.
Section 5.4 is about creating informative test automation reports.
Differentiation between test automation metrics and system under test (SU) related metrics.
Responsibility for managing SU related metrics lies with the project's overall test manager.
Test automation metrics track the progress of test automation efforts.
External and internal test automation metrics explained.
External metrics measure the impact of test automation on testing activities.
Internal metrics assess the effectiveness and efficiency of the test suite.
List of external test automation metrics provided, including automation benefits and effort to build automated tests.
List of internal test automation metrics, such as tool scripting metrics and automation code defect density.
Explanation of automation benefits and its importance in reporting the value of test automation.
Benefits of test automation depend on the objectives set for the test suite.
Possible measures of automation benefits include efficiency gains and hours saved by automating tests.
Discussion on the balance between automated and manual testing methods.
Coverage amplification and early defect detection as benefits of test automation.
Unique defect discovery as a significant advantage of test automation.
Indirect benefits of test automation through enabling additional manual testing.
Conclusion on the importance of measuring the benefits of test automation for a comprehensive understanding of its value.
Transcripts
welcome back to Tae certification
training Series today we are about to
start a new chapter in our test
automation Journey so far we have
covered a lot from why we automate test
to getting ready for its execution and
exploring the complexities of a journal
test automation framework now we are
about to dive in something equally
important but different which is test
automation reporting and
metrics in plain terms this chapter is
all about the information and numbers we
can gather from our automated test it is
like a dashboard in your car when you
drive you need to know how fast you are
going how much fuel you have and whether
the engine is working well or not
similarly in test automation we need to
keep an eye on various indicators to
ensure that everything is on track we
will explore test automation reporting
and metrics through four key
sections the first one is 5.1 selection
of T metrics this section focuses on
choosing the most relevant metrics for
your test automation
project then we have 5.2 implementation
of measurement in this section you will
learn how to put measurement into action
and gather valuable data
effectively next is 5.3 logging of the T
and and the Su in this section you will
discover the importance of comprehensive
logging in both the T and the
Su and finally 5.4 test automation
reporting in this section you will
explore how to create informative
reports that turn data into actionable
insights for your automation
efforts so let's start with Section 5.1
selection of Tes metrics and discover
how to choose the write metrics to guide
your test automation
Journey this section focuses on metrics
for monitoring the test automation
strategy and the Effectiveness and
efficiency of the
T so it's primarily concerned with
discussing metrics used to assess and
keep track of two main things mentioned
over here the first one is test
automation strategy which is about
monitoring how well the planned approach
for automating test is working and the
second is Effectiveness and efficiency
of the T it involves evaluating how
effective and efficient The Chosen
automation strategy is in achieving its
intended goals and
objectives next we have separate from Su
related metrics used to monitor the
system under test and the functional and
non-functional testing of the Su which
is selected by the Project's overall
test manager so the key Point here is
that these automation metrics are
distinct and separate from another set
of metrics known as Su related metrics
these Su related metrics are used to
monitor this Su which is system under
test which is the software or
application which is being tested along
with its functional and non-functional
aspects and the responsibility for
selecting and managing these Su related
metrics lies with the Project's overall
test
manager test automation metric allow the
Tam and Tae to track progress towards
the goals for test Automation and to
monitor the impact of changes made to
the test automation
Solution on the other hand the test
automation metrics are the domain of the
Tam test automation manager and the Tae
test automation engineer these metrics
are used to track the progress of the
test automation effort and to evaluate
how changes made to the test automation
solution are impacting its performance
so to summarize test automation metrics
are like the instruments for gauging the
efficiency of the automation strategy
while Su related metrics focus on the
quality and performance of the software
being tested each set of metrics serves
a distinct purpose and is managed by
different roles within the testing
team next we have t metrics and there
are two of them external and
internal so the external T metrics are
used to measure the T impact on other
activities in particular the testing
activities and the internal one used to
measure the Effectiveness and efficiency
of the TS in fulfilling its
objectives think of it this way external
metrics show how your Tas affects the
bigger picture of testing while internal
metrics tell you how well the TS is
doing its job job it's like looking at
both the forest and the trees to
understand the whole
picture so let's have a look at all the
external Tes metrics over here and then
later on we will go through all of them
one by one in detail the first one is
automation
benefits number two effort to build
automated
test three effort to analyze automated
test test
incidents four effort to maintain
automated
tests five ratio of failures to
defects six time to execute automated
tests seven number of automated test
cases eight number of pass and fail
results nine number of false fail and
false pass
results and number 10 code coverage
and similarly we have a list of internal
TS metrics too the first one is tool
scripting
metrics number two automation code
defect density and the third one speed
and efficiency of T
components let's now have a look uh at
all the external T metrics one by one
and the first one is automation benefits
why mayor automation benefits it is
particularly important an to Mayor and
Report the benefits of a
t this is because the costs in terms of
the number of people involved over a
given period of time are easy to
see people working outside testing will
be able to form an impression of the
overall cost but may not see the
benefits
achieved think of your TS as a toolkit
you have carefully assembled you have
put in time and effort to make it work
seamlessly now it's time to reveal the
real trayers hidden within this toolkit
which is the benefits of test automation
now here's the deal while it's quite
clear how much effort and resources go
into setting up testing automation like
the time and people involved the
benefits might not be immediately
visible to everyone especially uh those
who are not directly involved in the
process that's why we need to shine a
light on these benefits making them
crystal clear for everyone to
see benefits depend on T
objectives any mayor of benefit will
depend on the objective of the
T typically this may be a savings of
time or effort an increase in the amount
of testing performed breadth or depth of
coverage or frequency of execution or
some other Advantage such as increased
repeatability greater use of resources
or fewer manual
errors so the reward you reap from your
TS depend on what you aimed to achieve
in the first place did you set out to
save time enhance testing coverage or
reduce errors the benefits align with
your original
goals possible measures include
efficiency gains number of hours of
manual test effort saved so count the
hours saved by automating test compared
to manual testing
efforts streamlined regression testing
reduction in time to perform regression
testing so highlight how much faster
regression testing becomes with
automation enhanced test execution
number of additional cycles of test
execution achieved track how many
additional testing Cycles you can
execute thanks to
automation expanded testing scope number
or percentage of additional test
executed measure the extra test you can
perform due to
automation test automation ratio
percentage of automated test cases
related to the entire set of test cases
although automated cannot easily be
compared to manual test
cases okay imagine you have a group of
test cases some of these test cases are
automated While others are manual now
you want to know what portion of your
total testing effort is done through
automated test cases from a pool that
includes both automated and manual test
however here is the catch automated and
manual test cases are like apples and
oranges they serve different purposes
and are used in different situations so
directly comparing them is not
straightforward so while you can
calculate this percentage to understand
the balance between automated and manual
testing in your process remember that it
doesn't tell you which is better or more
effective it's a simple way to see how
you are using these two types of testing
methods then we have coverage
amplification increase in coverage
requirements functionality
structural so assess the broader
coverage you achieve whether it's
related to requirements functionality or
structural
aspects then we have early defect
detection number of defects found
earlier because of the T when the
average benefit of defects found earlier
is known this can be calculated to a sum
of prevented costs so calculate the cost
savings by finding defects earlier in
the development process thanks again to
automation then we have unique defect
Discovery number of of defects found
because of the T which would not have
been found by manual testing for example
reliability defects in this one identify
defects uh uncovered solely because of
automation which might have been missed
in manual
testing additional consideration note
that test automation generally saves
manual test effort this effort can be
devoted to other other kinds of manual
testing for example exploratory
testing defects found by these
additional test can also be seen as
indirect benefits of the T as the test
automation enabled these manual tests to
be executed without the T these test
would not have been executed and
subsequently the additional defects
would not have been
found here is the exciting twist test
automation not only saves time and
effort but also opens up opportunities
for more comprehensive testing and
deeper exploration it's like having a
versatile toolkit that allows you to
explore different testing scenarios and
uncover issues that might have otherwise
stayed hidden in the end measuring the
benefits of test automation is like
revealing the true value of your testing
efforts it's about acknowledging the
efficiency accuracy and Agility it
brings to your testing process and
inspire you to continue leveraging its
potential for even more impressive
outcomes this concludes today's session
we will continue exploring the remaining
TS metrics in our next session until
then take care
Ver Más Videos Relacionados
Павло Сафонов - Як оцінити ефективність автоматизації
ISTQB FOUNDATION 4.0 | Tutorial 58 | Benefits and Risk of Test Tools | Test Tools | CTFL
CompTIA Security+ SY0-701 Course - 5.4 Summarize Elements of Effective Security Compliance.
ISTQB FOUNDATION 4.0 | Tutorial 52 | Test Monitoring & Test Control | Test Metrics | ISTQB Tutorials
ISTQB FOUNDATION 4.0 | Tutorial 49 | Test Pyramid | Testing Quadrants | Test Management | CTFL
0204: Coverage measurement
5.0 / 5 (0 votes)