How to Measure The Efficiency of Testing

As software products become more complex, the pressure on testers to deliver products free from bugs has risen to higher and higher levels.

Unfortunately, limited resources, tight schedules and dealing with distributed teams have made it harder for testers to face the challenges involved and make sure they keep on top of their game at the same time.

Test metrics are a helpful way to measure the efficiency of the testing process. Analysing the correct metrics will reveal a lot of ideas on how to improve workflow and devise better strategies.

Measuring testing efficiency is also vital when an organisation chooses to outsource testing and would like to find out how well their investment has been working out.

In this blog post, I present some of the most common metrics used to measure the efficiency of testing, and point out why most of them are flawed and that the one, critical metric isn’t what most people think it is.

Common testing efficiency metrics used

So, how do we go about testing the testers?

Some of the most popular testing efficiency metres used to evaluate QA teams include:

Number of bugs found
Number of automated test cases written
Percentage coverage in functional testing
Number of bugs found in different stages of development

Which metric is the best to measure tester efficiency?

Just like LOC is a terrible metric to judge a developer by, number of bugs found or tests written are useless, especially since these metrics may be more directly related to poor development rather than having an efficient testing team.

Automation metrics are fine, however these are usually more helpful for developers than testers.

Which means that, ultimately, customer satisfaction is the best measure of testing efficiency for the whole team.

This can be described in a more elaborate manner as: the number of bugs found by the users that haven’t been detected by testers. A related metric which would also reflect efficiency would be the severity of the bugs that users report.

The most important metric is determined by your customers

Yet, it has absolutely everything to do with what happens at your offices while your testers are busy excavating layers of code to unearth issues and resolve them.

In fact, the more bugs your testing team can find before the product is released, the more efficient they will be.

The crucial metric is therefore how close to bug-free you can deliver software to your users.

This is seemingly a very unfair situation, since there’s no way of knowing how efficient a tester is until a user stumbles across a bug and reports it.

Basically, you have to be constantly aware that your efficiency scorecard will be determined by what happens in the next minute, and the next, and the next, most probably by a total stranger who could is based half-way across the world.

There’s no room for smugness if you’re tester. Every moment could potentially be the one where you come crashing down from your pedestal if you so much as take your testing efficiency for granted.

An inevitable drawback is that there could be a considerable delay between finishing your work on a project and learn how well you performed.

Summary

Building a good track record of finding bugs before release takes time, as well as a lot of effort invested in keeping your testing skills sharp. That’s why earlier this year I launched my first e-book, which is packed with ideas to help testers simplify their work and become more valuable team members.

There’s a tonne of information out there about metrics that help you master true efficiency and effectiveness as a tester.

However – just like in any old business – it’s the customer’s opinion that really counts, which is why it’s so important to have a system in place that instantly and easily captures user feedback and lets you extract valuable insights about the quality of your work.

For more information about how to improve your efficiency and effectiveness as a tester download Ulf Eriksson’s FREE 40-page e-book. In his book, Ulf shares 10 tips and strategies from the trenches that he learnt over a career spanning 20+ years. Download it now!