Work Smarter and More Effectively with “Reality” Acceptance Testing

Note: The content of this blog post first appeared on Konsultbolag1’s website on 8April 2016. This is an English translation and adaptation of the original article in Swedish by Charlotta Carlsson. Used with permission.

Have you already been involved in acceptance testing and realised, to your dismay, that the end-results didn’t turn out what you had expected?

Maybe you’re caught in a routine where you feel like you just keep repeating the same tests that have been performed over and over by others. Or maybe the job wasn’t done properly the first time round, and the acceptance tests missed out on important details and errors that should have been detected sooner.

In any case, if you want your organisation to start experiencing real business benefits that can take it to the next level, then you must find a way to enhance the acceptance testing process.

Our friends at Konsultbolag1 have created a model of acceptance testing they call “reality” acceptance testing, which can be readily used by many software development organisations that use agile management techniques.

In this article, we give an overview of this method of performing acceptance testing that you and your team can use to work smarter and more effectively.

A simple model for acceptance testing

To achieve acceptance for real, it is not enough to do acceptance testing on a project – or departmental-level. Instead, it has to be an ongoing effort that pulls in resources from every role on a team in a holistic manner.

A modern acceptance process, for example, should try to bridge the boundaries between requirements and testing aspects of a project. It’s really all about accepting a final working product and not simply testing something for its own sake.

The acceptance process is an integral part of the software development lifecycle, and plays a crucial part in determining when a product is ready to be launched.

What follows is Konsulbolag1’s simple model of acceptance testing, which can be used with any kind of development methodology.

Definition

The first step is defining what type of results one should achieve in a project or by the product overall to qualify for acceptance. This is the basis of the acceptance process, and the definitions of the results should take into account these three areas:

Effect
Quality criteria
Test ideas

Besides formulating concrete objectives for the project (effects) and having clearly-defined quality criteria, it is important to have test ideas, also commonly known as acceptance criteria. This final element is what ultimately makes any requirement testable.

Acceptance criteria are doubly helpful because they provide everyone in the team with a common understanding of the requirements needed. The process involved in developing these acceptance criteria is in itself a testing activity that typically occurs prior to the specific functionality being developed, although it is possible to develop test ideas after development is underway.

Requirements

Here is an example of the development of an administrative system where the requirements are expressed in the form of who-what-why, with some corresponding test ideas:

Who will be using the system? The payroll administrator.
What will they do with it? Seeing records of vacation time or sick leave taken, as well as the part-time schedule of staff members.
Why will they use it? To be able to respond quickly to part-time staff who want to know how much vacation time they have left.

Test Ideas:

Being able to see all the details of an employee on a single screen without scrolling. The viewport will be at least 24 inches wide
Automatically calculate the the number of vacation days left after specifying which calendar days are holidays
Having the ability to clear past choices and make new calculations
Receive a warning message when the number of calendar days selected exceed the total vacation time allowed.
Having the ability to calculate different scenarios for a particular employee, and to record actual data from a vacation form
Having the ability to save future vacation time that has been booked

Testing each one of these ideas will increase our understanding of what kind of results users are expecting from an app, as well as to guide developers in building a functioning model. By examining these test ideas, it is also possible to estimate how long the project would take, as well as how much it may cost.

It is important that these questions are raised and answered as early as possible in the project.

Responsibility

Every team needs to encourage a responsibility culture where everyone contributes to the development of the product according to their roles. This can be achieved by adopting the “reality” acceptance testing perspective from start to finish.

Here are some examples of acceptance responsibility based on different roles. Naturally, the roles will look different in different organizations.

If you are a developer, system architect, or in any way responsible for the operating environment, then your thoughts about the nature and quality of the requirements available are valuable to the acceptance process. If you think the requirements aren’t clear enough, for example, it’s important that the rest of team is aware of your misgivings.
As a product owner or client, you’re representative of the users and therefore need to be involved in the development process to communicate the project vision and explain what is required of the system in order to do its job properly.
Testers must try to stay a step ahead by providing feedback on the requirements and prototypes. They should participate actively throughout in the entire development lifecycle, not just at the end of it.
As a requirements analyst, you work together with both technical and business people to ensure that requirements are testable.
As a security officer, you work together with product owners, technical and business people to capture and describe the risks associated with the requirements, as well as develop test ideas about security.
Make sure to involve users throughout the development process of the project.
If you’re the team leader or acceptance test manager, then you also have a special responsibility to promote a responsible and communicative culture.

Behaviour

The effectiveness of acceptance testing depends a lot on how much the members of a team are willing to contribute towards making the process work. It also depends on how well the system is capable of supporting user behaviour that directly aligns with the product goals.

These are some ways we use to manage both aspects of behaviour:

Encourage open and ongoing communication, preferably face-to-face and in a casual setting.
Business and development need to remain in close contact throughout the requirements gathering and the development period. It’s a good idea to from an acceptance team that includes members from both sides.
Involve real users. The smaller the project, the greater the likelihood that end-users can be included in the acceptance team.
Test continuously from an acceptance perspective.
Evaluate and celebrate successes!

Outcome

With an effective acceptance process you can expect several positive outcomes, such as:

Shorter lead times
A product that fulfils requirements
Better control of time and money spent

Challenges for the acceptance process

Switching to a new model of acceptance testing may sound easier said than done.

It’s likely that an organisation will initially run into some difficulties while coming to terms with the reality of re-defined roles and procedures, and new systems and supplier relationships. This makes it harder to break old patterns.

If you recognize yourself and your organization in the following situations, you are not alone:

Clients are not closely involved with the development team
There is a complex systems environment that involves several integrations
Not everybody wants to contribute to the requirements gathering

If it is not possible to apply our acceptance model fully in your organization, you can still use it as an ideal to benchmark against. This way, it can help clarify your organization’s dilemmas and help you make the most of the opportunities you have.