Risk can be defined as possibility of an adverse or unwanted result or occurrence. If stakeholders, users or customer opinion about the project’s quality or successful completion of the project can potentially be reduced due to a problem or issue, risk is said to exist.

If the main impact of risk is on quality of the product, the problems are termed product risks, quality risks or product quality risks.

Table of contents

1. What is Risk Based Testing?
2. How to identify risks?
3. How to assess risk?
4. How to mitigate risks?
5. How to manage lifecycle risk?
6. Project Risk Management

If the main impact of the problem is on success of the project, the problems are termed planning risks or project risks.

A common challenge in test management lies in correct selection of a limited set of test conditions from an almost unlimited set of tests, then allocating resources, effort and prioritizing the tests effectively.

After the test conditions are chosen, the team must allocate requisite resources to create test cases, then decide a sequence for the test cases so that overall test efficiency and effectiveness is optimized.

Test Managers can use risk analysis to decide on appropriate test cases and their sequencing but there exist many variable factors and constraints that must be taken into account, even if it means compromising on test solution.

What is Risk Based Testing?

In risk based testing, the selection of test conditions are guided by the risks identified to product quality. These product quality risks also are used to determine the allocation of effort for the testing the test conditions and for prioritizing the created test cases.

Many types of testing techniques – depending on documentation and formality level – are available to perform risk based testing.

The main aim of risk based testing is to minimize quality risks to an acceptable level.

Achieving zero quality risk is close to impossible.

While carrying out risk based testing, the product risks or quality risks are detected and reviewed during risk analysis of product quality with stakeholders.

After risk analysis, test team carries out test design, test implementation and test execution activities with the objective of minimizing the risks.

Here, quality refers to all the product features, characteristics and behaviors that have the potential to affect satisfaction experienced by users, customers and other stakeholders.

When defects are identified before product release, testing has reduced quality risk by identifying the defects and providing methods to handle them.

Here are some examples of quality risks:

• Delayed response to users action: Non-functional risk pertaining to performance
• Reports with wrong result or incorrect calculations: Functional risk pertaining to precision
• Complicated UI and input fields: Non-functional risk pertaining to usability & system adoption

If defects are not found after testing, testing has reduced quality risks by confirming that the system performs correctly under tested state.

Risk based testing has several techniques for implementation which differ in the documentation level, type of documentation and formality level.

There are four main steps in risk based testing:

1. Identifying risks
2. Assessing the risks
3. Mitigating risks
4. Managing risk

The activities listed here are not sequential but overlapping. In the subsequent sections we will look at all these activities in detail. The cost incurred in these activities are part of the cost of quality.

For maximum efficiency, the team undertaking risk assessment and identification must comprise members from all stakeholders groups – be it the project on the whole or product development team.

However, in reality, some stakeholders take on the responsibility of representing some additional stakeholders.

Let us consider a scenario: When software is being developed for mass market, a small representative sample of prospective customers could possibly assist in identifying those defects which might affect the extent to which they use the software the most.

Here, the small section of prospective customers is the representative for complete customer base.

The testers should also be a part of risk identification process because of their experience in performing quality risks analysis and defect identification.

How to identify risks?

The stakeholders can use any methods given below to identify risks:

• Interviewing experts
• Making independent reviews
• Existing templates of risks
• Retrospective meetings in projects
• Conducting risk workshops
• Brainstorming with all stakeholders
• Creating and using checklists
• Revisiting previous experiences

If widest possible cross-section of stakeholders is used, the risk detection process would be able to detect a large majority among the important product quality risks. Stakeholders role in risk based testing is quite significant.

Risk identification process also delivers other results, i.e. they identify issues that are not really risks to the quality of the product. For example, generic concerns of the product, issues related to documents like requirements specifications etc.

It is important to manage project risks for overall testing, it should not be limited to risk based testing.

How to assess risk?

After risks have been identified, their assessment can start. Risk assessment is the analysis and evaluation of identified risks.

Risk assessment usually involves these activities:

• Classifying each risk
• Identifying probability of each risk occurring
• Impact of each risk
• Identifying or assigning risk properties like risk owner

Risk is classified on the basis of different parameters like performance, functionality, reliability, etc.

Companies are adopting ISO 25000 standard which replaced ISO 9126 standard quality characteristics for classification.

Many companies also used other classification models to classify the risks. The checklist used in identification of risks is usually used in classification of risks as well.

There are many general checklists available freely, which are customized by the organizations for their own use. If checklists are used for risk identification, risk classification is also done simultaneously.

To find the risk level of a risk, you need to ascertain the probability of that risk occurring and its impact when it happens. Probability of that risk occurring implies the probability of existence of the problem in application while it was being tested.

Likelihood can also be determined by evaluating the technical risk level.

Issues affecting this likelihood or probability include:

• Both technology being used and teams being employed
• Training level of business analysts, project managers, architects and developers
• Level of disagreement among team members
• Physically spread teams across the country/globe
• Old versus modern approach
• Testing tools and techniques
• Strength of leaders – technical & managerial
• Non-availability of previous quality assurance reports
• High rate of change
• High rate of early defects
• Problems in interfacing as well as integrating

The impact of a risk when it occurs is the importance of its effect on all stakeholders like users and consumers.

Some of the factors that influence product as well as project risks include:

• Rate at which the risky feature is used
• How crucial the feature is for achieving business objective
• Loss of reputation
• Damage to business
• Probable losses or liabilities in terms of finances, ecology or societal pressures
• Probability of criminal or civil legal sanctions
• Annulment of license
• Safety
• Non-availability of feasible workarounds
• Negative publicity because product failure is prominent

Risk level can be evaluated qualitatively or quantitatively.

If the probability of a risk and its impact is calculated then these numbers can be multiplied to get a risk priority number which is quantitative.

However, usually risk level can be determined only qualitatively. So probability of a risk occurring can be called very low, low, medium, high or very high. But a percentage value for the probability cannot be calculated to any degree of precision.

In the same way, impact of a risk can be very low, low, medium, high or very high but it cannot be depicted in financial numbers. However, qualitative evaluation of risk levels should be taken to be below par as compared to quantitative methods.

In fact, incorrect use of quantitative evaluations of the risk levels can result in misguiding the stakeholders regarding level to which the risks are actually understood and can be managed.

Risk analysis which isn’t based on broad and statistically validated risk data will be based on stakeholders’ perspectives.

The stakeholders – project managers, architects, business analysts, programmers and testers – tend to have their own subjective view of the risk probability and its impact.

So they will have different and at times highly varied opinions about each risk.

The process of risk analysis must incorporate some method of reaching a consensus or at least an established risk level. This prior agreed level could be through management order or by calculating mathematical figures like mean, median and mode.

The risk level should also have a good enough distribution in the given range, so rate of risks can provide correct guidelines for deciding on sequence, priority or effort allocation for test cases. Else, risk levels will not be able to act as risk mitigation guideposts.

How to mitigate risks?

The first step is analysis of quality risks i.e. identification and then evaluation of risks to product quality. All test plans are based on this analysis of quality risks.

Test designing, test implementation and test execution is done to mitigate the risks as per the test plan. The effort allocated to developing, implementing and then test execution is directly proportional to the risk level.

This implies that thorough techniques like pair wise testing are designed for higher level risks while not so detailed techniques like exploratory testing for limited time or equivalence partitioning are designed for lower level risks.

Development and execution priority of a test also depends upon risk level.

Risk level should also influence these decisions:

• Should the project artifacts and tests documents be reviewed?
• How independent should the testers be?
• What should be the experience level of the tester?
• How much retesting should be done?
• How much regression testing should be done?

While the project is continuing, some extra information may change the quality risks that the test team was working with or impact level of the risks.

Test team must always be alert to such information and tweak tests as and when required. The adjustments like detecting fresh risks, re-evaluating risk levels, assessing efficacy of risk mitigation tasks completed, etc. must be done at key milestones of the project.

For example, if risk detection and evaluation session was held on the basis of requirement specification in the requirements phase, it is advisable to re-asses the risks after design specification is complete.

Let us look at one more example. If number of defects in any part of the product is much higher than the anticipated amount of defects during testing, the testers can safely summarize that the probability of a defect occurring in this region is higher than anticipated.

So, the probability of risks must be revised upward for that part of the product.  This would also raise the extent of testing to be done for this part of the product.

Risks in product quality can be minimized even before execution of test cases begins.

For instance, if the issues concerning the requirements were detected during risk identification itself, the issues can be mitigated just after the process through reviews.

As this is done before subsequent phases in product development lifecycle, it will reduce the number of tests needed during subsequent quality risk testing process.

How to manage lifecycle risk?

Ideally, managing risk is a continuous effort through the complete product development lifecycle. If available, documents pertaining to test policy or test strategy must explain the following:

• Process to be followed for managing tests for project as well as product risks
• Integration of risk management to all test levels
• Impact of risk management to associated test levels

In an experienced organization, the project team is highly aware of risks and committed to managing risks at multiple levels, not only for testing.

In such a scenario, critical risks are dealt with at all test levels, as soon as they are identified.

For instance, in case of performance being a risk factor for product quality, testing of performance is performed at multiple levels like design testing, unit testing along with integration testing.

Experienced organizations do not stop at identifying the risks. They also identify the source of risks as well as the consequences.

Often, root cause analysis is performed to understand risk source in depth and apply improvements in processes to avoid the defects from occurring. Risk mitigation is performed all through the lifecycle.

In mature organizations, risk analysis takes into account the following factors:

• Related work activities
• System behavior analysis
• Risk assessment based on cost
• Analysis of product risk
• Analysis of risk related to end user
• Analysis of liability risk

In such organizations risk analysis goes beyond software testing. Testing team proposes and becomes part of risk analysis covering the whole program.

Most of the risk based testing techniques combine some methods to utilize risk level for sequencing and prioritizing the tests.

In this way they also ensure that most of the important defects are identified at the time of test execution and most of the essential components of the product are also covered.

Sometimes, all tests that are high risk are executed before any low-risk test executes and that too in the order of risk, starting from the highest risk. This is also termed depth first risk testing technique.

Sometimes a breadth first testing is used, i.e. a sample group of selected tests is run across all the predefined risk areas. This ensures that all risk areas are covered at least one time.

Depending upon the test process adopting breadth-first or depth-first technique, the time allotted for the testing process might be over before all tests could be executed.

After doing risk based tests, the testers report to the management about the risks levels that remain to be tested.

This information helps the management to confirm if they want to do more testing or not.

If more testing is not done, the remaining risks must be handled by the customers, end users, technical support staff, help desk and operational staff or a combination of one or more of them.

When the test is being executed, the risk based testing enables project manager, product manager, senior managers and other stakeholders to observe and manage the development lifecycle.

Keeping a tab on the development cycle enables them to take decisions regarding product release, depending on the remainder risk levels.

However, to allow the stakeholders to monitor the status, the Test Manager must provide risk based testing results in a way that is easily understandable.

Project Risk Management

Planning for project testing must also cover potential risks associated with a project. The procedure for identifying such risks is explained above in the section on identifying risks.

The detected risks must be communicated to the project manager so that he/she takes steps to mitigate them as much as possible.

The test team may not be able to mitigate all the risks but these risks can be taken care of:

• Testing environment readiness
• Testing tool readiness
• Availability of well-qualified testing staff
• Unavailability of testing standards, techniques and rules

Managing project risks includes the following:

• Organizing testware
• Testing the test environments before they are actually used
• Testing preliminary product versions
• Using difficult test entry conditions
• Strict adherence to testability requirements
• Being part of review teams for preliminary project work products
• Managing changes to project based on defect detection
• Reviewing project progress and product quality

After risk identification and analysis, these are the four ways to manage risk:

1. Preventive measures that decrease its occurrence and/or its impact
2. Creating emergency plans to handle the risk if it actually occurs
3. Transferring the responsibility of risk management to third party
4. Ignoring or accepting the risk

Some of the factors that should be considered while choosing the best possible option out of these four include:

• Advantages and disadvantages of an option
• Cost of implementing an option
• Extra risks related to choosing an option

In case of contingency plans, risk trigger as well as plan owner must be predetermined.

If you are interested, you can review Risk Based Testing from the perspective of the Foundation Level exam.

In the next topic we look at the various Risk Based Testing (RBT) Techniques.

Other popular articles:

• What is Risk based testing?
• What is Project risk in software testing?
• What are the different Risk Based Testing (RBT) Techniques?
• What is Product risk in software testing?
• How to Assess Quality Risks in Agile methodology?