Reviews can be considered to be a type of static tests, so Test Managers could be made accountable for the success of review process, especially for testing work products. This accountability should be fixed by the organization as part of its organizational policy or test policy.

Reviews find a mention in the syllabus for Foundation Level as per ISTQB in the form of static testing tasks for software products. You should note that management reviews and audits lay emphasis on software development process and not on its work products.

Table of contents

1. Review leader
2. Examples of reviews
3. Management Audits and Reviews
4. Managing Reviews
   1. Skills of a reviewer
   2. Factors affecting review planning
   3. Review leaders responsibilities during review
   4. Review leaders responsibilities after review
5. Reviews Metrics
6. Managing Formal Reviews

Review leader

In any organization formal reviews are done before as well as during software development projects. So, Test Manager, QA Manager or Review Coordinator may be responsible for the review process. The person responsible for the review is referred to as review leader.

Review leader is responsible for creating a favorable environment for successful reviews. A review leader must develop metrics to measure business value provided by the process.

Product testers understand software requirements and functional features of the final product well, they must be involved in any review process.

Training must be provided to all reviewers so that they clearly understand their role in the review process. It is essential that all reviewers must be dedicated to ensuring a successful review.

If reviews are done in the right manner, they prove to be the single largest and best cost-effective technique for improving quality of the final product. Therefore, it is essential the review leaders incorporate effective reviews in projects as well as prove its advantages too.

Examples of reviews

These are some of the items that may be reviewed in a project:

• Review of contract
• Project requirements – Functional as well as non-functional
• Top level designs
• Detailed design
• Code reviews
• Test work products like test plan, test criteria, risks to product quality, test data, test results, test environments, etc.
• Test entry and exit at each level
• Product acceptance in the form of approval by customer/stakeholder

Each of these reviews should be initiated as soon as the necessary documents are available for review, so that review results may be incorporated as soon as possible.

As you can see, a product undergoes multiple reviews at different stages of development.

However, the review leader should ensure that besides the static testing, some form of dynamic testing as well as other types of static testing should also be done to enhance coverage of tests as well as detect more defects.

Combination of testing techniques should be used because each technique has a different focus and ends up detecting different type of defects.

Consider the example of requirements review, which can detect issues at the project inception stage so that it does not get transmitted to the coding level.

Static review can assist in ensuring adherence to coding benchmarks and detect problems which can be difficult later on. Reviews can also train the team members in avoiding defects in the work products created by them.

Syllabus of ISTQB Foundation discussed these review types:

• Walkthrough
• Informal review
• Inspection
• Technical review

Besides these review types, Test Managers may also do audits and management reviews.

Management Audits and Reviews

Management reviews help in tracking project progress and assessing project status. They also provide support for taking decisions about further course of action like changing project resources, incorporating corrective activities, modifying project scope etc.

Management reviews primarily:

• Are conducted by the managers that are directly responsible for the application or project or by a skilled reviewer on their behalf
• Are also done by a decision maker like director or any other stakeholders themselves, or by a skilled reviewer on their behalf
• Check the real time project progress against planned progress for consistency
• Evaluate risks to the project
• Assess effect of various actions and how to measure the effect
• Create list of issues that need to be fixed, decisions that need to be taken, and actions that must be carried out

When management review of processes are done, they end up improving the overall process.

It is necessary for the Test Managers to be part of management reviews and even start the review process sometimes.

Audits are conducted to ensure compliance to certain standards, regulations, or contractual requirements. They are always independent to any other review or testing process that the organization or project may have.

Here are some of the main audit features:

• Audits are usually done and/or led by a principal auditor
• Compliance is tested by sourcing and analyzing documents, interviews, and witnesses
• Work product of audits include recommendations, observations, corrective measures, success/failure evaluation etc.

Managing Reviews

Reviews must be scheduled at natural milestones that occur in the project. Usually they should be done after requirements gathering and top level designing has been done, starting from objectives for the business as well as going up to the lowest design level.

Management review may also form part of verification done after, during and before testing as well other important project phases. It is essential to integrate review strategy with test strategy.

Review leader must consider these factors before creating a project level review plan:

• Processes and work products to be reviewed
• People involved in review process
• Risks to be covered

Review leader needs to recognize the project areas that must be reviewed in the early stages of project planning. The suitable review type which could be walkthrough, informal, technical or inspection and formality level must be specified as well.

Depending on type and formality level, review leader can ask for a training of reviewers, depending upon the budgeted time and money.

The budget must evaluate risks that are covered and ROI (return on investment).

Review ROI is the variation in cost incurred in carrying out the review versus the cost of handling defects detected later in the lifecycle assuming the review was not performed. This figure can be obtained as per the calculation for cost of quality discussed under business value of testing.

Required time for the reviews can be estimated based on these factors:

• Format of content which is to be reviewed
• Skill level of the reviewer
• time frame for delivery of reviewed item
• Time needed for review

The review leader pre-determines the metrics for evaluating review effectiveness at the test planning phase itself. In case of inspection, it is to be performed as and when sections of the documents are completed, on author’s request.

Test planning must also define review objectives like performing effective review, achieving maximum efficiency, and reaching unanimous decisions.

Often, the whole project is reviewed, requiring reviewing of project subsystems as well individual software units. Type, number, reviewers, complexity, etc. of reviews is decided by project size, product complexity, and probable risks.

Skills of a reviewer

To achieve required efficiency levels, the reviewers must have these skills:

• Technical knowledge
• Operational knowledge
• Mindful of the details
• Exhaustiveness
• Ability to provide clear and actionable comments
• Understanding of their own role and responsibility

Factors affecting review planning

Review planning needs to take care of these factors:

• Technical risks – Appropriate technical knowledge.
• Organizational risks – Sufficient time given to each reviewing organization, participation from review team members for discussions.
• People issues – This includes availability of skilled reviewers, commitment by each review team, and backup plan for replacing reviewers if needed.
• Time constraints – Adequate time provided for training.

Review leaders responsibilities during review

These are the things review leader needs to ensure during the review:

• Sufficient data points are captured in the review for evaluating review efficiency
• Checklist generated and maintained for enhancing future reviews
• Each detected defect has a severity and priority associated with it and defect severity and priority are defined before the review

Review leaders responsibilities after review

These are the actions review leader must take after completion of review:

• Gather metrics generated by the review
• Detected issues fixed to meet review objectives
• Determine ROI on reviews using metrics generated
• Offer feedback to reviewers
• Share review results with relevant stakeholders

Test Managers should compare the review reports with the real results obtained in tests done after the review to measure usefulness of the reviews conducted.

If there is discrepancy in the two results, the review leader must investigate and analyze why the defects could not be detected during the review. Some of the reasons could be:

• Incorrect entry and/or exit conditions
• Unsuitable team members
• Insufficient and inaccurate tools for review
• Inadequate preparation time
• Less time scheduled for review meetings

If a pattern is found in defects that were left undetected, this points to issues in the review process itself. In this scenario, review leader must investigate the causes and fix the problems.

Reviewing of review process and fixing of errors needs to be taken up if the reviews start losing their efficacy over a period of time.

Whatever the scenario, review metrics values should be utilized for identifying issues in processes, not reprimand reviewers or people whose work introduced the defects. Test Managers should be able to motivate the testing team to perform the review to the best of their abilities.

Metrics for Reviews

Review leaders are responsible for providing metrics for:

• Review team quality evaluation
• Review costs assessment
• Assessment of downstream benefits of reviews conducted

Metrics may be used in to calculate ROI, review efficacy, creating reports, and suggesting process improvement.

The below metrics may perhaps be evaluated as well as reported in case of every work product:

• Time required to prepare for review
• Size of work product
• Time taken in review process
• Time required for fixing defects
• Defect severity and how many defects were reported
• Areas having greater occurrence of defects
• Review type
• Defect density
• Density estimate of defects left undetected

Each review should evaluate these metrics and report on them for evaluation of the process:

• Effectiveness of detecting defects
• Effort and time required for review
• Percentage of review done as compared to planned coverage
• Defects types of detected defects with their severity
• Survey measuring perception about review efficacy and impact
• Number of reviewers
• Relation between type of review and types of detects detected
• Cost of quality measurements for defects caught in the review as compared with testing and product defects found later
• Savings in project time
• Average effort needed for each detecting each defect

The metrics discussed here are for evaluation of the product. However, they can be used to evaluate the processes as well.

Managing Formal Reviews

Any review has these phases:

• Planning the review
• Initiating the review process
• Preparation and Training
• Review meeting and discussion
• Rework
• Follow up

Review leaders must ensure adherence to all these steps for review success.

These are the characteristics of formal reviews:

• Pre-determined exit and entry conditions
• Availability of review checklists
• Deliverables like review summary, evaluation sheets, reports etc.
• Metrics for providing information about review progress, efficacy, and effectiveness

Before a review is started, the review leader must ensure that prerequisites for conduction of the review is met. If these conditions are not met, review leader can choose to refer to a more competent authority for any of these:

• Redefining of review objectives
• Remedial actions to start the review
• Postpone the review

In order to manage formal reviews, the above reviews are observed from the higher perspective of the program. So they are linked to project quality assurance. They need to be kept under constant observation and feedback about product and/or process metrics must be produced.

In the next topic, we explore defect management in detail.

Other popular articles:

• What are the roles and responsibilities involved during a review?
• What is Review process support tools?
• What is Formal review?
• How Do Software Development Lifecycle Activities & Work Products Affect Testing?
• What is Technical review in software testing?