You can’t work Agile without automated testing

For many people who work with agile development, simply automating the component tests and/or unit tests is the full extent of automation. Meanwhile, at the levels of integration testing and system testing, automation is conspicuously absent, which results in many tests being performed manually. Of course this makes it very difficult to have time to do enough regression tests.

One reason why so little of the testing is automated is because the system is prone to change more often when the work is agile. If automated tests are created without careful thought and planning, they will break when the system changes and maintaining the test cases will be unnecessarily costly.

How can I create a sustainable architecture for automation which works well in agile development?

Most people who work with Agile know that automating system testing is essential, but they also realize that it is very easy to fail in doing so. What makes test automation so hard within the agile context is that the test object is constantly changing since it is gradually refined over sprints. If there is anything a test automation engineer is afraid of, it’s constant changes and the resulting high level of maintenance.

Despite these difficulties, the hard truth is that without automated testing, it is not possible to work Agile in full, at least not with frequent high quality deployments.

The reason it’s so important to automate tests is that the number of test cases at the system testing level will continue to grow on par with the new functionality added in each sprint. This increases the total amount of functionality and code to be tested in every sprint, as new code will be added while the old will still need to be regression tested. This is different from component testing, which focuses almost exclusively on the functionality of the current sprint.

Simply put, automated tests are the only way to achieve a sufficiently high test coverage in each sprint and thus provide the high quality and rapid feedback that we seek when we are working Agile.

An agile project without test automation is essentially just a waterfall project in phases. It is simply impossible to satisfactorily regression test in every sprint if the tests are performed manually.

The result is often a postponement of regression tests and deployment to a regression test period made after every three to six sprints. This causes important feedback and bug reports to reach the team very late, even up to six months after the functionality was developed and that is exactly the same way as in a phased waterfall project!

What should be automated first?

When a test automation strategy is developed, it is easy to think that you should automate all regression tests, and that it should be done immediately. Of course this is not economically justifiable; it is not advisable to automate everything right away because manual testing will continue to play a major role.

It is important to determine both what is worthwhile to automate and what should be automated first to get as large gains possible by automation.

There are many factors which determine what should be automated first. Some of the candidates which typically fall in line first for automation include:

• Business-critical functionality
• Functions that are used frequently by many users
• Configuration testing where tests will be run with different configurations
• Test cases which will be run several times with different test data or conditions
• Test cases where it is easy to see the expected result.

Is it profitable to automate?

Convincing management to add resources and make an investment in test automation will require making an informed ROI calculation showing whether and when the investment can be financially profitable. In order to complete the calculation, we need to know the cost of automation.

A good default value is that it takes 4-8 times as long to automate a test case compared to the time required for manual testing. Therefore, the automated test case needs to be executed for 4-8 iterations for the investment to break even. Tests that need to be run just a few times or very seldom are not profitable to automate.

Even tests that are difficult to automate should be omitted. In addition to the cost of writing and maintaining automated test cases there are also several “hidden” costs such as licenses, infrastructure, support and training.

What do I gain from test automation?

The benefits of automation are various. The time saved when executing tests automatically instead of manually is an obvious advantage. Another advantage is that test automation can lead to more exhaustive testing because with automation, nothing stops you from executing the same test several times with more varied test data and perhaps even test different environments.

However, the main benefit of test automation is that confidence in the system and its quality is increased when more comprehensive tests are performed. This trust makes it possible to improve and customize the system to the needs as required in an Agile project.

Additionally, resources become available for other tasks. Instead of retesting existing functionality, testers can put their energy into testing areas of new functionality where a human feeling is really needed. Developers also become more confident with the help of automation. There will be fewer errors left as developers quickly see the consequences of a code change, thus saving the team’s (and your organization’s) time and money!