Collecting Feedback from Beta Users

In an announcement we sent out a few weeks ago, we revealed that our team is hard at work on developing a brand-new grid for ReQtest. Our call for beta users was met with a flurry of excitement, as many people in our user community signed up to test-drive the new and improved system.

Developing a better grid

At ReQtest we are developing a new grid to modernize and to be able to develop features we have been longing for but haven’t been able to, due to technical restrictions and limitations in the component we are currently using.

The grid is very central, you could call it the heart of the tool, because very many features are reached from here. We wanted to make sure not to miss requirements from daily needs of ReQtest users.

As our team continues developing our new grid using the valuable information our user community is feeding us, I took some moments to jot down my reflections on the process of collecting feedback from beta users.

This is how we do beta testing at ReQtest

We decided to do beta testing early (already from the very first beta) to use the feedback also for prioritizing. For our own beta process, we recruited around 20 users who volunteered to try the new grid and offer us feedback about their experience of using it.

At first, I was not sure this would be useful. What could be tested at this early stage? What will the beta-testers think of this very slim version? Would they find it useless and loose interest? Could we really expect any feedback at all?

The developers emphasized they wanted to go ahead, and I’m happy I agreed in spite of my initial concerns. This was really good stuff!

This is how we do it:

1. Before you start – Three decisions to guide your planning

a) Decide on what you want to achieve. Make sure you know what information is relevant and how you are going to use it. Otherwise don’t go ask for it!

b) Decide on how to collect the feedback. Be prepared to handle whatever and to answer to everyone who is asking questions or sending requirements or ideas.

Use a tool such as ReQtest. You can set up a specific e-mail address in ReQtest to collect feedback and convert it automatically into requirements or bugs. I think this is very convenient, as you can compare the new input with your already existing requirement.

c) Decide on what to do with the feedback. Beta testers expect you to keep your word and take their feedback into account in your work, otherwise they’ll feel like they have been cheated of their time and effort. Stick to your roadmap; it is tempting to do other fancy things. Take action that fits with your goals and avoid the temptation to simply change the goalposts because of an impulse to explore in another direction.

2. Selection of participants

We wanted to have a mix of beta testers in different job positions (testers, test leaders, product owner and other roles) to cover different business needs.

We selected around 20 “new” and “old” customers because the old ones could have gotten used to the ReQtest-style of doing things but we also wanted to hear fresh impressions from new users. We looked for a variety of users in different industries, public service, manufacturing, service, large companies and small ones.

3. Communication with customers

We chose to send an e-mail to people who matched our criteria, asking if they were interested in doing this together with us. And, they were! They were attracted by the possibility to help and to impact.

Besides mailing invitations to selected users, we also informed on our website that we welcomed beta testers and that anyone interested could send a note of interest to join.

After signing up, all the participants received an e-mail confirmation describing what to expect during the beta testing. We added a screen shot in the mail to show them what the grid will look like on screen and how to access the beta.

A note about promising new features – Don’t promise too much. It’s important that the beta testers have the right expectations, otherwise you will end up with disappointed users looking for features that never made it in the release and they’ll lose their trust in you.

In the e-mail there were also a few questions/issues we wanted them to give extra attention. We included information about how to send us their feedback and provided them with contact information in case they wanted to ask the team further questions.

4. Collecting feedback

We decided to handle the feedback as requirements. We keep them in ReQtest to be able to compare them with other requirements we have got already. The feedback can then be used to write requirements but also to change the order of prioritization in the backlog.

Sooner or later, you have to decide whether to develop or not. When feedback is all kept in one place, you will be able to handle the decision making that takes place during and after beta testing in a smooth, effectively and more holistic way.

5. How to prioritize feedback

Maybe there are couple of features you don’t think is that important but everyone is longing to use and test. In ReQtest you can quickly classify and filter features according to “Business value” and “Cost” to help you compare features and decide how to prioritize them.

It’s important that once you draw up a list of prioritised requirements and start implementing them you keep the developers informed about any changes. If the order changes too much, it might mean extra work and increased time and money costs. You should always ask yourself whether changing plans is worth it or not at this stage.

6. Release-handling and communication

Before the release, get back to the testers and remind them how valuable their feedback was and point out the concrete ways it helped you.

Inform them about what they will be able to do in the next release, before it is actually released. It’s just basic courtesy and it will also add to the feeling of pleasure that comes from being part of a beta testing cohort. If you treat your recruits well they’ll be more likely to help you the next time round.

Remember to thank everybody again after the new release is launched and encourage customers to keep sending in their feedback.

Conclusion – Back to the grid

Talking about loops, we’re not done yet! Our team is actively collecting data from our current beta testing and when that’s over… it will start all over again until next release, and then the release after that.

As always, the aim is to create a tool that really supports the needs of the users in every way possible.

Stay tuned for more news about our new grid. We’re expecting an official announcement and release any minute now!