System and acceptance testing – To automate or not to automate?

System and acceptance tests are expensive. Tests are carried out manually, they take a long time to complete, the work is monotonous and it is difficult to repeat the tests in the same way every time. However, system and acceptance tests are critical, so they must be carried out; there is no choice to be had here. Regression tests will need to be designed to verify that the system works after maintenance and new development and also to validate the system using feedback from customers and users.

The big question at hand here is whether or not to automate your system and acceptance tests.

First, a bit of the history of test automation

Testing tools for automating tedious tasks have been readily available for many years. Some such tools include TestComplete, HP QuickTest Professional, IBM Rational Functional Tester and Microsoft Visual Studio Test Professional.

Sadly, the results of the usage of these products are questionable. The first tools were based on coordinates so that the tester who created the test case clicked a button and the test tool stored the button’s coordinates in the test case. While tools like these are rather easy to learn, as soon as the button is moved, the test case needs to be updated. Later versions of the tools were given the ability to “see” graphical elements such as buttons, check boxes and tree menus, thus making test cases much more stable given that the tool does not break that easily when components are moved around during development.

However tools such as TestComplete, HP QuickTest Professional, IBM Rational Functional Tester and Microsoft Visual Studio Test Professional are also associated with a number of drawbacks. The tools are complex and large which makes them very expensive, not all graphical elements are supported by the tools and in cases where graphical elements are not supported, the tool falls back to using coordinates in the old-fashioned way, making test cases created this way highly vulnerable.

Owing to the above points, the test tool must be carefully evaluated and the system to be tested always kept in mind. This way the chances that the automated test tool really works with the specific system under test are greatly increased.

Automation at different levels

There is no single solution that will fit all the levels of detail of the system. On the lowest level, the code level, code review is a good solution. The components may be component tested by means of test frameworks such as Nunit and Junit.

On a higher level of detail, the graphical user interface can be tested with the help of tools for recording and playback.

Visual GUI testing

Today’s testing tools for automated system and acceptance tests are based on image recognition and scenario-based testing. This enables the tools to emulate a human user in a better way. Examples of tools are JAutomate, Sikuli, Eggplant, SeeTest, T-Plan Robot and Testing Anywhere.

JAutomate

JAutomate is a tool for automatic recording of test cases. Jautomate has the possibility to see hidden images and texts. You can do both automatic and semi-automatic test steps and the tool supports data-driven test cases. Test cases can be exported to FitNesse.

Sikuli

Sikuli is an open source tool that was developed at MIT. It uses Python as a scripting language. You can import Python and Java Library. The tool supports OCR (optical character recognition). You can export executable scripts.

Pros and cons of automating system and acceptance testing

The tools for visual GUI automation are independent of the underlying technology. Testing is more like a human user. It’s easy to understand and change the test cases. It does not require much technical knowledge to be able to use the tools.

Changing an organization costs money because it takes a long time to change the practices and employee attitudes. The system under test imposes limitations – when the system is slow to use, the tests also are slow. To administer the tests cost money.