One of the pains of being a test manager is that part of your job is to provide management with key, pertinent information about the testing process and the progress it makes. You probably have to do this rather regularly and as the process is ongoing, which doesn’t make it easier.

To do this and do it well, you need to choose the right metrics and after having chosen the figures which you feel or know are best, you also have to sell your message to your bosses, the decision makers. We’ll cover that bit in another article, and for now, let’s focus on selecting the key metrics which should affect your decision.

 Create Your First Test Case in ReQtest (Start your 10-day trial now)

 

A word about metrics

Many organizations use comprehensive (and costly) testing methodologies, but you’d be surprised how few of them track their investment so as to determine how ultimately profitable their testing strategy is. This is neither desirable nor wise, but it is a fact.

The fact is that there are many different metrics which can be used to gauge your organization’s success and cost effectiveness in testing. Some metrics show you whether your company tests thoroughly enough or sufficiently early; others measure the effectiveness of the testing you conduct; and yet others calculate how much money your company spends fixing bugs and other errors.

 

banner

 

Defect Detection Percentage

One of the metrics which measures the quality of your company’s testing is the Defect Detection Percentage. The formula is simple, tally the number of defects found during testing, then divide that number by the number of total defects and there’s your Defect Detection Percentage!

The reasoning behind the Defect Detection Percentage is that if users find most of the errors after a system is launched instead of your development and testing teams finding the errors prior to launch, your testing process has not been effective.

For example, if 80 defects were found in testing and 20 additional flaws were found in the two months following launch, congratulations, you have a Defect Detection Percentage of 80%, meaning you caught on to 80% of defects during testing. This is not such a good percentage to have, but hey, at least now you know!

 

Cost of Defects

A useful metric to have is one which gives you an average of how much each defect costs you. When you combine the cost of defects to the Defect Detection Percentage, your gains by raising your Defect Detection Percentage will be made clear.

Finding out the cost per defect is fairly simple. Choose a project and examine the last 30 defects in production. Find out which people were involved in debugging and bug fixing, and how much time each activity took. This might look something like the below:

The longer your decision chain is, the higher your total figure will be.

Next, multiply the number of hours by the hourly rate your organization pays each role. It might be best to use the figure your company uses as a cost basis. Assuming an hourly rate of $150, 20 hours of labour means that every bug found in production costs $3000 to fix.

The next step is to make the corresponding calculation for the other levels of testing and requirements, from requirements management through component testing and acceptance testing. You can use this as the basis for calculating how much defects cost at each level.

After this, calculating gains made by changing a test process so that more defects are found sooner is a rather straightforward matter.

 

Defect Leakage

No software testing methodology is so efficient that it can discover all the errors and bugs from a system. In fact, even after rigorous testing, some bugs will remain in the system, hidden or undiscovered, only to be uncovered later on in the software development life cycle. Defects like these are known as leakage, and there are a few ways to find defect leakage.

One very commonly followed approach for finding defect leakage is the matrix method in which you will have to analyze all the defects and determine in which stage or test level the defect was discovered. If there are large gaps in your findings then this might indicate that your testing efficiency is rather low.

Another approach you can take is that of using a defect tracking system (such as ReQtest) and force whomever is logging bugs to include the detection phase whenever a bug is logged. By following this approach you can easily define bug slippage as well as the areas in which improvement is needed.

Of course, at this point it is necessary to clarify that each metric answers different questions, therefore, the process of selecting metrics should start with determining which questions you want to answer. Things you might ask include “Do we put the right amount of time into the right steps?”, “How much do defects really cost?” and “How effective are our testing processes?”

The formulation of your objectives should be SMART: specific, measurable, accepted, realistic and time-based, and to learn how to set SMART objectives, read this article to help you get started – http://www.reqtest.com/blog/newsletters/how-to-set-smart-goals/

banner

 

Author Ulf Eriksson

Ulf is the founder of ReQtest and as the Product Owner, decides what features are added to the product, and makes sure that ReQtest is of a consistently high quality. Ulf has written several books and courses as well as a library of articles on the subjects of testing and requirements management, as well as speaking publicly on a number of subjects related to the world of testing.

More posts by Ulf Eriksson

Join the discussion 2 Comments

  • Hello Ulf:

    Interesting article. Thanks. However, the last time I checked, $150/hr * 20 hours is $3,000, not $30,000. And if you’re looking at the incremental cost of finding the issue in production versus testing, you’d want to subtract out the cost of fixing the issue before production, which is presumably less than 20 hours, but not zero. You may also want to include the cost to the business of a production defect, which could be significant – but admittedly this is hard to quantify. Not all bugs are created equally.

    FYI, a big challenge I come across regularly is classifying issues as bugs or enhancements – just because a user doesn’t like something doesn’t mean it’s a bug (unless we’re also including requirements defects, which is another issue, often outside of the development/testing arena). And then you have to weed out issues that are reported as bugs that turn out to be something else – data issues, user errors, etc. As you correctly mention early in your article, many companies fail to track testing effectiveness. One reason could be that it can be quite costly to ensure the accuracy of the information. And bad metrics based on dirty data is I think worse than no information, and can lead to corrective measures aimed at the wrong problem.

Leave a Reply