Testing Metrics Which Concern Decision Makers

One of the pains of being a test manager is that part of your job is to provide management with key, pertinent information about the testing process and the progress it makes. You probably have to do this rather regularly and as the process is ongoing, which doesn’t make it easier.

To do this and do it well, you need to choose the right metrics and after having chosen the figures which you feel or know are best, you also have to sell your message to your bosses, the decision makers. We’ll cover that bit in another article, and for now, let’s focus on selecting the key metrics which should affect your decision.

A word about metrics

Many organizations use comprehensive (and costly) testing methodologies, but you’d be surprised how few of them track their investment so as to determine how ultimately profitable their testing strategy is. This is neither desirable nor wise, but it is a fact.

The fact is that there are many different metrics which can be used to gauge your organization’s success and cost effectiveness in testing. Some metrics show you whether your company tests thoroughly enough or sufficiently early; others measure the effectiveness of the testing you conduct; and yet others calculate how much money your company spends fixing bugs and other errors.

Defect Detection Percentage

One of the metrics which measures the quality of your company’s testing is the Defect Detection Percentage. The formula is simple, tally the number of defects found during testing, then divide that number by the number of total defects and there’s your Defect Detection Percentage!

The reasoning behind the Defect Detection Percentage is that if users find most of the errors after a system is launched instead of your development and testing teams finding the errors prior to launch, your testing process has not been effective.

For example, if 80 defects were found in testing and 20 additional flaws were found in the two months following launch, congratulations, you have a Defect Detection Percentage of 80%, meaning you caught on to 80% of defects during testing. This is not such a good percentage to have, but hey, at least now you know!

Cost of Defects

A useful metric to have is one which gives you an average of how much each defect costs you. When you combine the cost of defects to the Defect Detection Percentage, your gains by raising your Defect Detection Percentage will be made clear.

Finding out the cost per defect is fairly simple. Choose a project and examine the last 30 defects in production. Find out which people were involved in debugging and bug fixing, and how much time each activity took. This might look something like the below:

The longer your decision chain is, the higher your total figure will be.

Next, multiply the number of hours by the hourly rate your organization pays each role. It might be best to use the figure your company uses as a cost basis. Assuming an hourly rate of $150, 20 hours of labour means that every bug found in production costs $3000 to fix.

The next step is to make the corresponding calculation for the other levels of testing and requirements, from requirements management through component testing and acceptance testing. You can use this as the basis for calculating how much defects cost at each level.

After this, calculating gains made by changing a test process so that more defects are found sooner is a rather straightforward matter.

Defect Leakage

No software testing methodology is so efficient that it can discover all the errors and bugs from a system. In fact, even after rigorous testing, some bugs will remain in the system, hidden or undiscovered, only to be uncovered later on in the software development life cycle. Defects like these are known as leakage, and there are a few ways to find defect leakage.

One very commonly followed approach for finding defect leakage is the matrix method in which you will have to analyze all the defects and determine in which stage or test level the defect was discovered. If there are large gaps in your findings then this might indicate that your testing efficiency is rather low.

Another approach you can take is that of using a defect tracking system (such as ReQtest) and force whomever is logging bugs to include the detection phase whenever a bug is logged. By following this approach you can easily define bug slippage as well as the areas in which improvement is needed.

Of course, at this point it is necessary to clarify that each metric answers different questions, therefore, the process of selecting metrics should start with determining which questions you want to answer. Things you might ask include “Do we put the right amount of time into the right steps?”, “How much do defects really cost?” and “How effective are our testing processes?”

The formulation of your objectives should be SMART: specific, measurable, accepted, realistic and time-based, and to learn how to set SMART objectives, read this article to help you get started – https://reqtest.com/blog/newsletters/how-to-set-smart-goals/