Metrics help in measuring the current state of an activity or process. They help us set benchmarks and targets. Measuring where you are currently helps you establish the how much further you need to go in order to achieve your goals. Test Managers must be able to define, track, report test progress metrics.

What gets measured gets done is a common saying. So it can be safely inferred that if something is not measured it won’t be done. So it is essential to establish quantifiable metrics for testing.

Table of contents

1. 4 categories of test activity metrics
2. Test progress metrics
3. Product quality risks metrics
4. Defect metrics
5. Test metrics
6. Test coverage metrics
7. Test planning, monitoring and control metrics
8. Test analysis metrics
9. Test design metrics
10. Test implementation metrics
11. Test execution metrics
12. Test progress metrics
13. Test closure metrics
14. Test control activities

4 categories of test activity metrics

Metrics for software testing activities can be grouped into the following:

• Project metrics – They measure how well the project is moving towards its exit conditions. Examples include test case percentage that ran successfully, failed or were executed.
• Product metrics – They measure product characteristics like density of defects or degree of testing.
• Process metrics – They measure the ability of product development processes or testing. Examples include amount of defects which testing has been able to uncover.
• People metrics – They measure the skill levels and ability of team members or whole teams. Examples include adherence to schedule for test case implementation.

Any metric can be part of more than one category listed above. For instance, if a chart is prepared to record the rate at which defects are being reported every day, it can be a project, product or process metrics.

If no defects are reported for seven days, the project can safely be said to be moving toward exit condition.

If no more defects are found in the product, that is a measure of product quality. If a huge number of defects are detected in early phases of testing, it measures the test process ability.

It is very important to handle people metrics very carefully as they can easily be confused with process metrics.

If that happens, the whole testing process can fail and people may lose confidence in their managers as well as organizational capabilities. We will look at how to motivate testing teams and assess them against established metrics in upcoming topics.

ISTQB Advanced Level Test Manager exam deals mostly with project metrics. A number of these metrics measuring testing progress also measure process and product.

Metrics assist the testers in reporting test results and tracking testing progress consistently. Test Managers usually present these metrics at meetings attended by stakeholders of different levels.

As these metrics may be used to assess the overall project progress, it is necessary to be careful while determining what metrics must be tracked, techniques for preparing reports on the metrics and frequency of presenting the reports.

These are some of the point a Test / QA Managers should keep in mind:

• Metrics definition – Metrics that are defined, should be useful. Unimportant metrics should be ignored, keeping in view the four categories discussed above. All stakeholders must concur with the metrics definition to avoid any confusion when discussing measurements. Since a single metric might give the wrong idea, metrics should be defined such that they balance each other and provide the complete picture.
• Metrics tracking – Tacking, merging and reporting of results for the metrics must be made automatic to the extent that is feasible to minimize effort spent on these activities. Test Managers must observe if there are deviations from expected results and incorporate them into the report. If possible, causes of the deviations must also be mentioned.
• Metrics reporting – Metrics are reported to the senior management for project management. So, the report should ideally be in presentation form and highlight the important metrics values as well as evolution of metrics over a period of time.
• Metrics validation – Test Manager is responsible for verification of data and values being presented in the reports. Test Managers should also analyze it for correctness as well as trends being reported.

Test progress metrics

Test progress is observed based on these 5 factors:

• Risks to product quality
• Product defects
• Tests conducted
• Test coverage
• Confidence in test activities

Product defects, risks, tests and scope is normally reported in a predetermined format. If these metrics are correlated to predefined exit conditions, a benchmark to assess test effort can be developed.

Confidence can be measured subjectively or objectively using metrics like surveys and coverage.

Metrics that can be defined for product quality risks

• Fraction of risks that were fully covered by tests
• Fraction of risks where all of or at least some of the tests failed
• Fraction of risks that could not be tested fully
• Fraction of risks that were tested or sorted according to category of risks
• Fraction of risks that were detected post preliminary analysis of risks to product quality

Metrics that can be defined for defects

• Ratio of total number of defects detected to number of defects resolved
• Mean time between failure or rate of failure being reported
• Categorization of defects according to these factors:
  • Product components to be tested
  • Defect causes
  • Defect sources like addition of new features, regression, requirement specification
  • Test releases
  • Defect levels
  • Defect priority or severity
  • Duplicated or rejected reports
  • Time lag between defect detection and resolution
• Daughter bugs, i.e. defects caused by fixing another defect

Metrics that can be defined for tests

• Number of planned tests
• Number of implemented and executed tests
• Number of tests that were blocked, skipped, failed or successful
• Status, trends and values for regression as well as confirmation tests
• Ratio of planned testing hours to actual testing hours daily

Metrics that can be defined for test coverage

• Coverage of requirements and design documents
• Coverage of risks
• Coverage of testing environment or configuration
• Coverage of product code

Test Managers must be proficient in interpreting and using the coverage metrics for reporting on testing status. Coverage of design and requirements documents are needed for higher testing levels like integration testing, acceptance testing and system testing.

Coverage of codes are required for lower testing levels like unit testing and component level testing. Higher level testing results should not include code coverage.

It is necessary to note that despite 100% coverage goals at lower levels, defects will be detected at the higher levels and fixed accordingly.

Testing metrics may be related to the primary testing activities. This will help in monitoring test progress against the stated project objectives.

Metrics that can be defined for test planning monitoring and control

• Scope of test basis elements like risk, product requirements, etc.
• Defect detection
• Ratio of estimated number of hours required in test development and execution to total number of hours required

Metrics that can be defined for test analysis

• How many test conditions were known?
• How many defects were detected?

Metrics that can be defined for test design

• Fraction of test conditions that had test cases
• In the test design process (for example, when test basis was used to develop tests), how many defects were detected

Metrics that can be defined for test implementation

• Proportion of testing environments that were setup
• Proportion of records of test data that were uploaded
• Proportion of test cases that were automated

Metrics that can be defined for test execution

• Percentage of test cases that were run, successful or failed
• Ratio of testing criteria covered by test cases that were run successfully
• Ratio of expected defects to actual defects that were reported or resolved
• Ratio of estimated test coverage to real coverage that could be achieved

Metrics defined to observe progress of testing

Metrics defined to observe progress of the testing activities must map to the project milestones, test entry conditions and test exit conditions. Some of these metrics could be:

• Number of predefined testing cases, conditions or specifications that were executed, with their results (pass or fail)
• Detected defects categorized according to severity, importance, affected product component, etc.
• Details of modifications required and their status incorporated and/or tested)
• Estimated versus real cost
• Estimated versus real testing duration
• Estimated dates for project milestone testing versus the real dates
• Estimated milestones dates of testing activities versus their real dates
• Details of risks to product quality categorized into mitigated and unmitigated
• Key risk components
• Risks detected subsequent to test analysis
• Loss in testing time and effort because of unexpected or planned events
• Status of regression testing and confirmation testing

Metrics to measure test closure tasks

Following metrics may be designed for measuring tasks involved in test closure:

• Number of test cases
  • For these categories – run, passed, failed, skipped and blocked
  • That became part of reusable repository of test cases
  • That were to be automated versus the real number of cases that were automated
• Number of defects that were resolved or not resolved
• Number of archived work products of testing

Test process is also monitored through other management techniques like work breakdown structure. Products developed using Agile techniques monitor testing process on burn down chart. In Lean management, test progress is monitored by creating a testing story card on Kanban board’s column.

For a group of metrics that are predefined, reports may be generated verbally, in form of tabular data or visually as graphs and charts. The metrics values obtained can be utilized for many things, like:

• Understanding causes of defects and identifying trends, if any
• Creating reports for project management team and other stakeholders
• Modifying testing process if needed and monitoring the execution of modified tests

Here there is no one correct way to collect and analyze the metrics or create reports using them. It depends on project goals and requirements and type and skill level of people using the reports.

The metrics gathered through the testing process must assist the Test Manager in monitoring the testing effort and leading it towards successful completion.

Therefore, the metrics, amount and frequency of data collection, its complexity and risk associated with it must be established in the planning phase itself.

The project conditions can keep changing during software development lifecycle.

Test control activities

Test control must be able to modify the tests according to changing project environment and data provided by the tests.

As an example, consider the scenario where dynamic testing of a product reveals cluster of defects in areas that had been assumed to be defect free, the time available for testing is reduced due to delay in development. This would require revision of the risk analysis and the plan. In this case, tests would have to be re-prioritized and the effort allocation for testing would have to be revisited.

Keeping in view the new information, test plan should be revisited, new test cases created and testing effort redistributed.

If reports on testing progress point to deviation from test plan, test control activities must be performed to steer the test towards the correct path.

Some of the actions that may be considered for this include:

• Modifying test sequencing or test plan
• Reviewing quality risk analysis
• Increasing testing efforts
• Rescheduling product release date
• Changing test exit condition
• Modifying project scope as required

All stakeholders and project managers must agree before any of these steps could be implemented.

Data provided in any test report depends on the intended audience. So, a report for project manager would have detailed defects report whereas the report for business manager should focus on product risks.

In the next topic we will explore the cost of quality.

Other popular articles:

• How to perform test prioritization & effort allocation in test process?
• What is test status report? and How to report test status?
• Best Test Management Tools – Web based
• What is test monitoring in software testing?
• What are the 7 principles of testing?