Exploratory Testing – Where Do Its Pitfalls Lie?

In recent years, the term ‘Exploratory testing’ became one of the more popular buzzwords in the testing world. The benefits of exploratory testing are many but success is not automatically guaranteed. As with most other things, if you have a good knowledge of the pitfalls of exploratory testing, you are more likely to succeed.

In this blog post I will go through some pitfalls and challenges of Exploratory testing based on my own experience and describe how they can be solved or avoided.

First off, what exactly is Exploratory testing?

In exploratory testing testers try to constantly adjust their tests based on how the software reacts during the tests instead of using pre-defined test cases. Using this method, tests will become much more dynamic than when test cases are used, resulting in more errors being found.

Exploratory testing can be compared to a game of tennis. When two people play tennis they don’t try to plan in detail how to proceed further in the game but they have a rough idea of how the game should be played and it’s the opponent’s behavior which partly controls how they play their game.

The same applies to exploratory testing. Only a loose idea of how testing is done is prepared before the actual test session. After that it’s the software’s behavior that controls how the tester performs testing.

We have a bunch of articles about exploratory testing on this blog; go ahead and check them out.

Challenges and Pitfalls of Exploratory testing

One of the major challenges of exploratory testing is the lack of attention from the organization and that it is easy for the tester to sometimes feel forgotten.

While a more traditional tester will often review the specifications, write test cases based on them, execute test cases and write bug reports, a tester who works with exploratory testing will not write and execute pre-defined test cases. Instead, the tester will perform tests while he or she is constantly trying to figure out what should be tested in the next step and how this should be tested.

Therefore, there is no simple and transparent way for the organization to get a quick overview of what the tester does. The only visible deliverable in this case is the bug reports. If no errors are found, no bug reports get written.

Lack of feedback / debriefing

Of course, during exploratory testing documentation is still written, but the documentation is very different from the traditional testing based on test cases. In order to increase traceability and show what has been tested, there should always be a test report. The report shall contain information on what was supposed to be tested and the purpose of the test session. Information on what was found during the test and what was not tested in the original test idea, and why this was not tested shall also be included. If the tester comes up with an idea that should be tested in future this is also documented in the test report, so that the test idea is not forgotten.

Here is one of the biggest problems with exploratory. Very few people in a company have the time or passion to read a test report where no obvious errors are presented. It is much easier to provide statistics on the number of written bug reports to know the quality. Right or wrong – this is the way many people work. If the test reports are not read, it is easy for testers to sometimes feel forgotten.

Getting credit for having executed all the assigned test cases before the deadline without finding very many errors is not uncommon. However, it’s not as common to receive appreciation for having performed exploratory tests for several days without finding very many errors. Although exploratory testing is often considered to be a very fun way to work with compared to for example testing based on test cases, exploratory testing risks getting boring after a while, if you do not work with a very changing software. After having learned the system and tested through most parts the work tends to be boring if no or very little entirely new functionality is added. When there are no detailed test cases it can be difficult to ensure that the tests are performed.

Misunderstandings about how exploratory testing is used

Many organizations believe they’re working with exploratory testing when they’re really only using the more or less aimless ad-hoc testing, or so-called “happy testing”. that is, testing with little control or monitoring. Although these types of testing can be considered a form of exploratory testing, the big difference is the lack of planning, management, documentation and follow-up. Exploratory testing assumes that there is a plan and an idea of how the system should be tested and what a session should include, even before the testing begins. When documentation and follow-up is minimal, the actual testing tends to become arbitrary. Therefore, it can easily cause resentment towards exploratory testing from both the organization and the testers, although they have not been working with real exploratory testing. Because of this, it is not uncommon to hear derogatory comments about exploratory testing from people who erroneously believed that they have worked with actual exploratory testing and failed.

The organization may have too high expectations of exploratory testing

Another problem may be too high expectations on earnings and the effect of exploratory testing, simply because the method has got a lot hype in recent years and in some quarters has been claimed to be the ultimate solution to all test-related problems that may exist. Organizations with poorly defined requirements, lack of cross-functional testing and unclear responsibilities for the various test and development teams can sometimes embrace exploratory testing to improve the quality of their product instead of getting to the bottom of their biggest problem areas. Done properly and with the right mix of exploratory testing and traditional testing exploratory testing can definitely solve some of these problems.

Another expectation that may exist is the ramp up time. Although it is often quicker to start a team working with exploratory testing than to kick off a traditional test team it does not mean that the team working exploratory will solve all problems and find all the bugs during the first week. Even a tester working with exploratory testing must be given time to get to know the software. Errors will be found during ramp up time, but depending on how used the tester is to the context, it may take a while to achieve high efficiency in testing.

Summary

Achieving exploratory testing that is both successful and fun requires a lot of work. The less interest from the organization, the harder the tester must work to get truly rewarding test work. In an organization with high test maturity and good insight into what it takes to get exploratory testing working you probably have to work less with getting attention from management.

When developers see that exploratory testing is faster when testing for example, bug fixes and hot fixes, chances are they will ask for more exploratory.

• Invite to short debriefings where test areas, test coverage, number of errors, and the general feeling of the software is presented.

• Define objectives, such as for the test session.

• Arrange unofficial competitions within the testing department where testers compete to see who can find the most serious bug, most errors, weirdest bug or similar. This can be a fun way to make testing more creative and increase community but it must not be perceived as a constraint.

The next step

Read our other blog posts on Exploratory Testing: