After detecting the defects, managing defects is the most important activity for any organization, not just for the testing team but for everyone engaged in the software development or project management process. The processes and tools used for managing the defects are critically important because they can identify the areas that need improvement for overall enhancement of development and testing processes.

Table of contents

1. The Defect Lifecycle and the Software Development Lifecycle
2. Defect States and Workflow
3. Managing Duplicate Defects and Invalid Defect Reports
4. Cross Functional Defect Management
5. Defect Report
6. Defect data
7. Assessing Process Capability with Defect Report Information
8. Examples of process improvements using defect report information

Test Manager or QA Manager needs to know which data must be collected at all costs and promote correct usage of processes and tools used for defect management.

The Defect Lifecycle and the Software Development Lifecycle

As you know, when anyone commits an error during the product development or maintenance, a corresponding defect is introduced in any work product. The defect may be introduced into any work product like requirements specification, technical document, use case, test case, code, etc.,

As defects may occur in any work product, defect detection and removal must be an integral part of every step of software development life cycle. Sooner the defects are identified and fixed, lesser the total cost of quality of whole system.

As discussed in the syllabus for Foundation Level, static testing process detects the defects directly, without the need for debugging. This reduces cost of fixing defects considerably.

In contrast, dynamic tests like unit testing, system testing or integration testing, defects are discovered only after they cause system failures. Once the tester observes the failure, a defect report is filed and further investigation into the defect starts.

In a Test Driven Development (TDD) technique, automated tests are built into the product design itself. The specified product development phase is not considered over till the automated tests are executed.

It must be noted that in TDD, tests will fail until the complete unit is developed. So there is no need to track failures till the unit is complete.

Defect States and Workflow

Many of the organizations that conduct software testing use a tool that keeps track of the defects throughout defect lifecycle and to manage defect reports.

Usually, there is one owner of the defects report at each state of defect lifecycle, who is also responsible for completing a task that would move defect report to the subsequent state.

In the last phases of defect lifecycle, defect report may not have an owner in these situations:

• The defect has been fixed and tested and hence defect report considered closed
• Defect report is cancelled because it is invalid
• Defect report is deferred if the defect won’t be fixed as part of the project
• Defect report is considered not reproducible if the defect cannot be observed any more

If defects are detected during testing, the testing team must take action to manage them in these three situations:

• Initial state – Also referred as new or open state. Here one / multiple testers are responsible for collecting necessary information for fixing the defects.
• Returned state – This is also called the rejected state or clarification state. Here the person receiving the test rejects the report and asks the report creator to give more information. Testers have the option of providing more information or accepting the rejection of the report. If many reports are rejected, Test Manager should look out for shortcomings in the preliminary information collection process itself.
• Confirmation state – Also called the verified or resolved state, here the tester does a confirmation testing to ensure that defect has really been resolved. This is done by repeating the steps that detected the defect during testing. The report is closed if the defect was fixed. The report is reopened if the defect was not fixed and sent back to the owner who previously owned the defect report, for fixing.

Managing Duplicate Defects and Invalid Defect Reports

The tester may detect an irregularity due to an issue in test environment, testware or test data or tester’s own mistake. If it is reported as a defect and later found to be unrelated to any work product defect, it is called as false-positive result.

The report is closed after terming it an invalid defect or directly cancelled.

Sometimes one defect can lead to results which may look like multiple unrelated issues to testers. If more than one defect is reported, which have a common root cause. In such situations, one defect report remains open while others defects are closed by marking them as duplicate.

Though duplicate and invalid reports indicate inefficiency, Test Managers must recognize them as inevitable. If they try to eliminate such reports completely, amount of false negative reports increases because testers are not willing to file reports for all defects detected.

This in turn decreases the organization’s testing efficiency.

Cross Functional Defect Management

Usually defect management is owned by the Test Manager and the testing organization. However, a team which is cross-functional, comprising members from product development, product management, project management, etc. is accountable for managing defects reported by test teams.

As defects are reported through the tools used for defect management, the committee for defect management must consider the costs, risks and benefits associated with each risk to decide whether the defect should be fixed or deferred.

If a defect needs to be fixed, its priority must be finalized with respect to other tasks in the project. The committee can consult Test Manager & the testing team members to discuss the priority and severity level of any defect.

It must be remembered that no tracking tool can take the place of proper communication. Similarly, defects committee consultations cannot be a replacement for optimum use of the tool used for tracking defects.

To ensure efficient and effective defect management, all these factors are necessary:

• Good communication
• Efficient usage of a defect tracking tool
• Sufficient support for the tool
• Clearly established defect lifecycle
• Proactive committee for defect management

Defect Report

When static testing detects a defect or static testing observes a failure, the testers must include them in their defect report. It is necessary for:

• Managing reports through each level of defect lifecycle
• Assessing project status in context of testing progress and product quality
• Assessing process ability

Information needed for managing defect report or assessing project status depends upon the lifecycle stage in which the defect has been reported. As the lifecycle progresses, more information is needed for defect report management.

Nevertheless, the core data collected must be constant across not just the current lifecycle but all projects so that process defect data comparisons can be done to get meaningful insights.

Gathering defect data helps in monitoring and controlling test progress and evaluating test exit conditions. For instance, defect data can provide insights into analysis of defect density, trends in defect detection and resolution, average time needed to fix a defect and intensity of observed failure.

Defect data

Information pertaining to defect data can include these:

• Name of tester who detected the defect
• Tester role, like developer, business analyst, technical support analyst, etc.
• Testing type that caught the defect – like Regression testing, smoke testing, usability testing etc
• Problem summary
• Problem description
• Testing steps to recreate the observed failure, including expected and actual results, screenshots, logs, database dumps, etc.
• Lifecycle phase where defect was introduced, detected and removed along with test level where required
• Work product where defect was initiated
• Severity of impact on system / stakeholder
• Defect priority (generally dependant on the issue’s business impact)
• Component or subsystem where the defect was found – for analyzing defect cluster
• Project task which was in progress when the issue was found
• Method of identification used that detected the defect – like regular production use, static analysis, review, dynamic testing etc
• Defect type and its taxonomy, if used
• Quality attribute impacted by the bug
• Test environment where the bug was detected, if applicable
• Product and project that has the defect
• Defect owner who at present has been allotted the issue
• Work products where defect was detected and where it was fixed. This should include release numbers, specific test item details etc.
• Defect reports current state
• Recommendation, approval and conclusion for corrective action performed or deferred
• Costs, benefits, risks and opportunities linked to whether defect is fixed or not
• Dates for various changes in the lifecycle of the defect. Report owner of each lifecycle transition / change. Activities undertaken to isolate and fix the defect and verify the solution.
• Description for defect resolution and details of how the fix can be tested
• Other references like tests performed to identify defects and test basis elements like risk linked to the bug

Test Manager can refer to many documents and standards like ISO 25000, IEEE 1044, IEEE 829 and Orthogonal Defect Classification to decide what data must be collected for reporting the defect.

Testers must enter complete, accurate, precise, timely and relevant defect information. Even manual intervention or correct communication cannot undo the effect if such defect information is not entered and it will affect project status evaluation, testing progress and ability level of the process.

Assessing Process Capability with Defect Report Information

Under Test Management, we discussed in detail how defect reports assist in monitoring project status and creating related reports.

Test Managers must understand the role of defect reports in evaluating software development process capabilities and testing.

Besides information used for monitoring test progress, discussed in Test Management as well as in Defect Report, information captured in the defect must support initiatives for process enhancement.

Examples of process improvements using defect report information

• Using information about phases in which the defects were introduced, detected and removed to evaluate phase containment while proposing how defect detection efficiency can be enhanced in all phases
• Using information about phases where defects were introduced in Pareto Analysis of phases where maximum defects were detected, so that targeted improvement measures can be taken to reduce defects
• Using information about root cause behind the defects to find the reason why defect was introduced, so that process could be improved to decrease overall amount of defects
• Utilizing information about phases in which the defects were introduced, detected and removed to carry out quality costs analysis and decrease cost of defects.
• Using information about defective product component to analyze defect clusters and understand the technical risks in a better way and help in re-building of defective components.

Sometimes, teams may decide to not track the defects detected for some or whole of the SDLC.

Though this is apparently done for efficiency and decreasing process overheads, it diminishes insights into capabilities of software development and testing processes. This also hinders process improvements recommended earlier due to dearth of trustworthy data.

We inspect software testing process improvements in the next topic.

Other popular articles:

• What is configuration management in software testing?
• What are incident reports in software testing?
• What is Incident management tools?
• How to communicate effectively as a Test / QA Manager?
• How to write a good incident report in software testing?