What makes a bug, what makes the bug a problem and how do I handle it?

Bug reports are one of the main work products we produce as testers. As for definition of a bug, I like James Bach’s definition: “A bug is something that bugs somebody who matters.” However, here I will focus on the ones that are found by investigating the product. This is not the only aspect though. I recommend that you watch Rob Sabourin’s Let’s Test keynote from last year. It will reveal just how flexible the definition of a bug can actually be.

It’s good to note that finding bugs is not the only thing I do as a tester. I also provide information that will help other people in making better choices. Recently, I wrote a blog post (Visibility of Our Work) that goes through the additional value we can provide as testers, besides finding bugs.

 

Let’s get back to finding bugs. When we find a bug and reach the point where it gets filed into the defect management system (or any other way of informing a programmer and possible other stakeholders about the bug), we have hopefully gone through many points that have a crucial effect on the probability of this specific bug being fixed.

BBST Bug Advocacy course slides mention a quote from the book Testing Computer Software (3.0) that highlights about how the amount of bugs we find, is not the main aspect that we should focus on.

 

The best tester isn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the right bugs fixed.

 

In this article, I will try to outline some of the phases that increase the probability of bugs being fixed, with respect to paying attention to figuring out if something is a problem (this relates to TCS quote above). In the end I’ll also show an example of what my bug report would look like. I will write a bug report of a bug I found on XMind (mind mapping tool – www.xmind.net).

Is this bug a problem?

A crucial phase that we need to go through before we can start filing a bug is that we need to decide if we have actually found a bug. In other words, we ask the question “Is it a problem?”

 

One way to answer this question is using oracles. An oracle is something that will help us to possibly recognize if there is a problem, for example, a requirements document. When we notice that the product behaves in a way that is not consistent with how it is defined in requirements, we decide that there is a bug in the product.

 

There are a lot of other oracles, such as live oracles, which might be a product owner, or an experienced user that will confirm to you that some specific observation is a bug.

 

Michael Bolton wrote a great blog post where he went through the high-level list of different oracles that we might use to recognize if there is a problem. The list is as follows;

 

a) A feeling like confusion or annoyance.

b) A desirable consistency between related things.

c) A person whose opinion matters.

d) An opinion held by a person who matters.

e) A disagreement among people who matter.

f) A reference document with useful information.

g) A known good example output.

h) A known bad example output.

i) A process or tool by which output is checked.

j) A process or tool that helps a tester identify patterns.

 

One important thing to remember about oracles is that they are fallible.  Let’s say, for example, I feel strongly that there is a problem, which is a threat to the overall quality of the product. Though, according to the requirements, the product is working correctly. I might still consult another oracle (like the product owner or someone else whose opinion matters) and explain how I feel that the current behavior is a threat to the overall quality of the product. If I will not succeed in convincing the stakeholders about the threat to overall quality, I have at least given my opinion.

 

After we have recognized and concluded, that we have a bug in our hands, we move on to reporting it (there are probably different ways to handle this, but personally I like to solve the “Is it a problem?” issue, before I report the bug).

 

It’s good to note though, that while some bugs can be reported instantly, others require more information gathering.

Isolate and amplify the bug

I often want to understand (isolate) what the bug actually is in its most understandable form, before I move on to reporting it or discussing it more with stakeholders. My goal is that the programmer can clearly envision the bug in their head when reading my bug report. To achieve this, we can try to place the programmer in the seat of the end-user, and try to make him or her see the behavior with the eyes of the end-user.

In my current project I am monitoring real time server logging when I test our system (logs are located on a Unix server and logging is done with the help of the tail -f command). In the logs I can often see information, which will help the programmers to troubleshoot the problem. They can also access the logs by themselves but I will do my best to save their time by providing events that I’ve witnessed in logs while testing. Let’s remember though, that the logs you provide are not necessarily related to the problem you have reported. This is a risk which needs to be acknowledged.

 

Besides isolating I also want to try to amplify the bug, if possible. As an example, some time ago I tested a message field that allowed text to be inserted. It also had a functionality which allowed you to attach files. I noticed that when you attach a PDF file to the message, the amount of characters that the system allows you to insert to the message field got smaller.  Further investigations revealed that if I increased the size of the file, the amount of characters available was decreased even more. Eventually, if I attached a large enough file, I was not able to insert text into the message field at all.

 

This information is good to mention when you report the bug. It will help in understanding the impact on the end-user in much greater detail.

 

We often understand more about the possible impact to the overall quality of our product when we have invested in increasing our knowledge about the bug. Furthermore, the more we understand the impact on the overall quality, the more we can describe the possible consequences of this specific bug.

 

In order to get the bugs fixed though, we need to explain our findings. Great testers excel in their ability to describe the behavior which they have witnessed. However, I’ll describe why testers need to be highly skillful at describing bugs in the next installment of this series.

 

About the author

My name is Aleksis Tulonen and I’m a software tester. You can find me on Twitter (@al3ksis) and I also write my own blog at flowoftesting.wordpress.com.

 Comiq is a software testing and quality assurance service provider located in Helsinki, Finland. Our skillful staff has years of experience in testing and QA of business critical IT systems. Our company web page is located at www.comiq.fi