Investing in software testing tools is a significant decision for many organizations. The Test Manager may be called upon to drive the selection / recommend a tool to be used. Selection of a testing tool should be done after due process, considering the long term strategy of the organization, total cost of ownership (TCO), risks and several other parameters, apart from meeting the functional requirements.

In this article and the subsequent articles, we discuss many of the aspects that a Test Manager needs to keeps in mind while selecting software testing tools and deciding about automation.

Table of contents

1. Tool Selection
2. Open-Source Tools vs Vendor tools
3. Developing Custom Tools

Tool Selection

Test Manager / QA Manager needs to study many issues when choosing testing tools. The usual course of action is to buy a test tool from an external vendor. Sometimes that’s the only option available. Otherwise, custom tools or open source are also viable options.

Irrespective of how the tool has to be sourced, the Test Manager needs to determine the Total Cost of Ownership (TCO) using a cost-benefit analysis for the entire expected lifetime for the tool. This issue has been explored in detail in the section on return on investment (ROI).

While one may think that the purchase of tools may add to the cost of quality of the project, tools generally improve the efficiency of the team and reduce the cost of quality in the long run. Tools can also make it easy to collect test metrics.

Open-Source Tools vs Vendor tools

Open-source tools can be used for all the phases of testing like test implementation, test execution, test monitoring and for activities like test case creation, defect management, testing automation, etc.

An important consideration in purchasing an open-source tool is that even though the cost of purchasing it is quite low, formal tool support is unavailable. Though, in some cases, devoted supporters of the tool may be ready to provide informal support.

Most of the open-source tools are developed to resolve a particular problem. This applies to open-source testing tools as well. So the Test Manager must consider carefully if the selected open-source tool would be able to fulfill all testing needs.

In comparison, vendor tools usually perform more activities than a corresponding open-source tool.

• An important advantage of using an open-source test tool is that it can be customized as per the organizational needs, by its users.
• If skilled people are available, two or more open-source tools can be integrated.
• Open source tools can be enhanced to meet testing demands or the original tool can be customized to address the needs.
• However, it must be ensured that open-source tools are not being used just for its sake because extending or customizing them adds testing overheads.
• Test Manager must ensure that ROI on using the open-source tools is positive.

Different open-source tools have different variations of the generic GNU General Public License. Some may require the developers to share their modifications with all other tool users.

Therefore, the Test Manager needs to understand the licensing terms and conditions thoroughly and legal repercussions of distributing the software after modifications.

• The quality of open-source tool selected must also be examined carefully, especially if software is critical to the mission or product safety needs to be tested.
• The precision of open-source tools are not generally certified but tools from vendors are usually certified for both relevance to task at hand (for example DO-178B) and correctness.
• Even if the open-source tool is good, certifying it may be left to the testing team that is using it, which would lead to additional overhead for the team.

Developing Custom Tools

Sometimes no open-source or vendor tool meets the organization’s requirements. This may be due to use of customized environment, proprietary hardware or customized processes.

If the team has the necessary skills, Test Manager might think of developing custom testing tool.

These are some of the advantages of having a custom tool developed:

• Meets the exact needs of the testing team
• Function efficiently as per requirements
• Can be designed to interact with additional tools
• Can produce data and output in a format which the team requires
• May be used for subsequent projects

These are some of the disadvantages of developing custom tools:

• Custom tools rely on the developers who developed the tool
• Tools need sufficient documentation necessary for maintenance by others
• Tool may be abandoned if there is lack of documentation, when the developers quit the project
• If scope is expanded in future, quality of the tool could be compromised, giving rise to defect reports that are false-negative, inaccurate data generation etc
• Like any other software, it must be designed, developed and tested in order to confirm that it works

We look at return on investment and one time/recurring costs associated with testing tools, the in the next topic.

Other popular articles:

• ISTQB Advanced Level Test Manager Study Material
• What is Testing Tool ROI? One time/Recurring Costs & Risks related to tools?
• How to manage software testing tool lifecycle and tool metrics?
• What are the important factors for the software testing tool selection?
• What are the parameters for selecting a testing tool?