3 Top Challenges Testers Face When Doing Exploratory Testing

These days, it is fashionable to say that your team practices Exploratory Testing. Using the term implies you understand the technique, and therefore, surely you occupy a rarefied sphere that few others can attain.

It’s one thing to know the technique, and a completely different thing to understand it well enough to be successful with it.

Frequently, I see Testers that have jumped on the Exploratory bandwagon without having first understood how they’re going to implement it. And not implementing Exploratory Testing right, or not preparing your stakeholders for it, can adversely impact the results, and more importantly, how others perceive the results.

By now, you are probably aware of Exploratory Testing and may be familiar with using it to augment your Test cycles. There is a lot of insight into the topic here on the ReQtest blog, if you want a refresher.

So, you’re ready to roll out Exploratory Testing as a standard practice in your organization. You’re super-keen on piloting the supersonic jet that will catapult your team’s efforts to another level. And just like any good pilot, you’d like to prepare for take off – by getting your plane inspected, by understanding the turbulence zones ahead.

Read on to understand the three major pitfalls of Exploratory Testing, and how you can escape them.

1 – Evidence

Traditional Test Cycles mandate a considerable lead time for up-front planning and preparation. We see Terms of Reference, Test Strategy, a Test Plan, and detailed Test Scenarios and Scripts all making their way in. Multiple reviews and iterations later, you’re ready to begin your Test Cycle with the support of these documents. Your team is able to communicate regular (daily) updates, demonstrating progress against plan. Progress meetings, defect triages and reviews happen at regular intervals, keeping sponsors informed (and hopefully happy) of the progress.

An Exploratory Test cycle, on the other hand, doesn’t provide nearly as much visibility.

To make matters worse, some testers assume they don’t have to document (you do, of course you do). To do that is to confuse Exploratory testing with Ad-hoc or Happy testing. Not documenting an Exploratory cycle leaves you with nothing to show for a session, or an entire cycle. And often, the only evidence of the effort is bug reports. Again, you could be testing a well-designed or well-tested part of the system, and there may not be many bugs to report. So what will you say you’ve been doing the past few days?

How it impacts you

In a world driven by productivity, not showing evidence for your efforts is bad for you and your team. Managers need to account for every penny spent on a project; and when they don’t know exactly what your team did during a days-long test cycle, accountability becomes an issue. The technique and the tester often get a bad rap for not showing ‘visible progress’.

What you can do about it

• Log as you execute: I’ve said this before – it is essential to log your test efforts, in whatever form, as you execute. Logging helps stimulate the next test idea, and simultaneously helps you create tangible evidence for your efforts.
• Share progress: One of the key differentiators for Agile are its burn down charts. The charts give project teams and their stakeholders a sense of progress and accomplishment. Therefore, communicate progress for your exploratory cycles – however you want to do it. For example, do you want to simply share your logs at the end of each hour-long session for all to see, take a break and get back to the next session? Go ahead and do it. As the log grows between sessions, so will everyone’s appreciation of the effort.

2 – Expectations

Do your sponsors understand what is Exploratory Testing and how it can help them? Do you understand? Educating your stakeholders about the technique, why you should use it, and how it can help your team is your responsibility as a Tester or Test Manager.

Why is it your responsibility? Because, anybody who is not a tester may not understand the significance of what you’re doing and how it is helping them.

How it impacts you

What they don’t understand, they can’t appreciate. More importantly, your stakeholders may develop impractical, unrealistic expectations of your Exploratory test efforts. There are instances of management assuming that exploratory tests will provide 100% test coverage. That is neither the intended purpose, nor is it feasible. Not seeing enough plan documentation, and consequently not seeing progress against plan, is unsettling to most managers, and makes them question the value Exploratory testing provides.

What you can do about it

• Educate: Introduce everyone to the concept, explain how it adds value to their lives. Specifically, explain how each role – developer, project manager, BA, sponsor – benefits from exploratory testing. Show how both Exploratory and Scripted testing complement each other, and work best when used together. Use examples of companies like Microsoft that use both on their projects.

3 – Timing

Are you on the last leg of a mandatory project that needs certification before submission to the regulator? Are you using Exploratory Testing reports to certify the system? At times, testers mess up by not analyzing what is right for a given situation. Knowing when to run Exploratory tests is more important than knowing how to run them.

How it impacts you

Exploratory tests aren’t made for situations that need complete test coverage or detailed auditable reports. For instance, the regulator won’t sign the dotted line if they can’t see concrete evidence. The sponsor won’t like it if you ran only Exploratory tests and missed a small but painfully visible copy error on the landing page.

What you can do about it

• Understand when to use Exploratory tests. Using exploratory testing up front in a project will help the team catch major bugs quickly and reduce impact to plan later. Run exploratory tests immediately after a bug fix to confirm effectiveness. Always finish a project with a scripted test cycle that checks for defects methodically – especially if the product is customer-facing, regulatory, or has anything to do with customers (e.g. staff system used to manage customer information).

Exploratory Testing doesn’t equal ‘no discipline’. Experience with the technique and understanding when to use it will help you succeed.

Do you agree with my top 3? What are your top 3 challenges when doing Exploratory testing? Comment below.