New requirements for testing in the mobile world

NOTE: This is a guest post written by Rebecca Holm and Hans Joelsson of Konsultbolag1.se that is being reproduced here on the ReQtest blog. 

Mobile phones have been with us for quite some time now, from the 80s brick-sized devices, to their widespread availability in the 90s and the mobile revolution of the 2000s, where we saw the explosion of SMS, the smartphone revolution and, with it, the advent of the mobile applications.

As long as mobile phones existed, so was there the mobile tester. Starting from the pure mobile tester who tested signal strength and SMS features, then came the early application testers who tested company-specific mobile solutions, followed by the modern mobile application testers who threw the market wide open to new and smaller companies with their ideas.

In the past, mobile devices and mobile applications were largely considered as being separate, disconnected from an organisation’s’ core activities. Nowadays they have become in many cases the dominant channel of communication with customers.

Naturally, there are many similarities between the testing of an application as an ordinary desktop program as a mobile application. However, there are certain significant differences that affect everything from the individual tester’s work, to how an organisation should plan for their mobile projects.

In this article, we will offer some examples of these differences as well as practical tips to consider when it comes to conducting mobile application testing.

Differences between PC and Mobile

In order to decide how to test a mobile application it can be helpful to know the differences between testing “normal” PC software and mobile applications. Some of the most important differences are the following:

PC

The testers usually sit at one place, their office;

WIFI or a permanent connection to the Internet is used;

The screen is usually not turned/moved constantly and lacks touch screen;

There is a higher tolerance by the target users, who expect the quality of the software to be higher for the reason that the delivery time is longer.

Mobile

More situations have to be simulated during testing, such as: poor coverage, standby, low battery, etc.;

Testing can take place anywhere as long as there is connectivity;

Testing must take into account different types of connectivity, e.g. 3G/LTE/WiFi/USB;

The application has to be tested in different screen modes, landscape or portrait;

There are rapid release cycles because the programs are smaller with limited functionality;

The tolerance is low among users since there are many other applications that do exactly the same thing. It is easy to select another application in the hope that it will be of better quality.

Challenges in application testing

The operating systems

There are at the time of writing three operating systems that dominate the mobile market: Android, iOS and Windows, and a host of smaller ones.

Android

Android is by far the largest operating system today. This is good news and bad news because the challenge with Android is that there are many active versions of the operating system being used concurrently. This situation came about because the latest version is not sent out to all who use Android in the same way as, for example, Apple does. In the latter case, the upgrade rate is close to 100% soon after a new version has been released.

Before Android 4.0 (Ice Cream Sandwich) which was released on 19 October 2011, there were five major versions in general use. Since then there has been continually updates as the chart below shows.

The latest version of Android called KitKat 4.4 and was rolled out between November 2013 and January 2014. Android 4.x Jellybean is the version installed on about 59% of all android phones worldwide.

Apple

As mentioned earlier, everyone who owns an iPad / iPhone has the latest version of Apple’s iOS, which makes it natural for users to update the software as soon as they get the opportunity. The latest version is in the current situation iOS 7.1.1.

Windows

The number of users running Windows on mobile devices is growing steadily and the most common operating system among this group is Windows 7.8. The latest version, Windows 8.1, however, is also the main focus for application development today.

Others

In addition to the leading operating systems, there are a few other smaller players, including Blackberry, Firefox OS and Sailfish. Blackberry is primarily geared towards business users while Firefox OS and Sailfish are newcomers that have generated significant interest among users. Who knows, maybe in five years, one of the latter two operating systems will dominate the market…

Units

The next challenge is that every manufacturer has several models of phones running different versions of the operating system.

Again, it is Android that accounts for the majority of the units since there are about 300 phones and 467 tablets. On the other hand, Apple has “only” 15 phones and 43 tablets, while Windows runs on about 40 phones and 221 tablets.

The reason there are such high numbers is because we counted each unit with its different memory variants and different opportunities for connectivity (WiFi, 3G, etc.).

To make it a bit more complicated, there is also a wide variety in the resolution of the devices, which may cause graphical errors depending on the device the application is running on.

 

Browsers

If you are testing a web-based (Responsive / HTML5) applications then you have to analyse its behaviour on each of the major mobile browser applications in use. The most common browsers are Internet Explorer, Firefox, Chrome, Safari and Opera, which means that a test suite quickly becomes very complex when tests are made on all browsers. In view of all this it is easy to start panicking whenever testing a mobile application.

Keep calm, there are tricks that can help you out

Before testing mobile applications, there are some questions that should be answered in order to make good decisions. The first concerns which platforms the application will support.

Today, it is common for applications to be first launched on iPhone and Android, but as Windows Phone is installed on more units, both supply and functionality on the latter devices will be on the same level as the first two. To determine the platform, one can start by looking at current statistics or logs from the company’s channels to make an analysis of which devices customers are using right now. If, for example, you find that the majority uses iOS, than that is the platform you should start developing for first.

If you want to launch an application on all three platforms, then you could test primarily on a flagship device from each operating system to keep abreast with the latest technology. It will be necessary to test on a sample of older devices and versions of the OS as well. These tests should be scaled down in most cases, otherwise testing will take too long: a situation rarely accepted by the client.

It is also good to keep updated on future phone release and updates of the operating systems in order to avoid unpleasant surprises. It is not unusual for a new phone or new OS to mess up an application’s functionality when it is released.

For those who want to cover as much testing as possible and keep the costs down, there are some mobile services that can facilitate testing. There are companies that offer beta testers who help to test, making the purchase cost of mobiles disappear. The advantage is that you can decide which phones and operating systems your application should be tested on and quickly get a wide testing coverage as there are over 15,000 testers in some networks.

We should not forget emulators for phones that can cover some of the tests, but it is not recommended to use emulators exclusively because they cannot simulate a real phone good enough.

Tips for testers

To facilitate the testing of mobile applications, we want to give the readers some useful tips to help them. The first is to combine exploratory tests with the tips below to make it easier to detect defects quicker and more effectively.

Simulate reality

As we mentioned at the beginning of the article, there are more scenarios to simulate when testing applications on mobiles compared to computers.

Trying to reflect actual events in everyday users’ life is a key factor in finding the bugs that show up, e.g. when the user is in the underground where the network signal strength is low. There are two general test areas where many errors tend to show up.

The first is when the user multitasks, for example, changing a song, googling a restaurant and check when the next subway goes.

The second is handling interruptions, where outages can include anything from lost coverage, the battery runs out, the phone goes into standby mode, or if the user receives a call.

Switch connection

It is important to try to switch from WiFi and 3G, and vice versa. It is easy to forget when you as a tester sitting in the office with the full speed of the wireless network that the users will mostly run on the mobile network.

Change language

It is easy to believe that the language you have on your phone shouldn’t make any difference, but there may be large differences in word length and special characters used in some languages.

One time when we tried to change the language, red lines showed up here and there in the application and the date was displayed differently which made ​​the whole day name partially invisible.

Combine different modes

One area that can generate amusing defects is a combination of idle/standby with landscape/portrait. This can cause peculiar looks, such as the following:

 

 

These are some examples of how it can look at the mix of different modes. It is sometimes best to lock the app in portrait mode to avoid these problems. But that is not always the best solution when the user may need to write or read a lot of information on the application.

Create a mobile testing strategy

Countless studies and reports show that large investments are necessary when it comes to the mobile testing.

Whether you’ve worked in mobile testing before or not, it is an area which you will inevitably get into, so it is important that testers understand its peculiarities.

As a business, it is also important to understand these extra dimensions and to understand that the old rule of thumb which says that testing should take about a third of the development period no longer applies. Overall, if we look at the areas that mobile test involves, the proportion n a mobile development project is closer to a 50/50 distribution between development and testing.

A first step to adapt to these new challenges is to create a mobile test strategy, or add a section in the current test strategy that addresses the specific needs of mobile testing.

Suggestions for things to include in this strategy:

Which operating systems, including versions of the operating systems will the application will be tested on?

Android X.X, iOSX.X, Windows Phone X, any others?

Which devices will the tests be performed on?

iPhone? Samsung? HTC?

Which networks and operators will be used in the tests?

WiFi/3G/LTE?

Local operators?

Operators abroad?

Roaming?

Will emulators, beta testers, cloud solutions be used?

Last but not least, the following two areas should also be considered in the test plan or strategy:

Performance

When it comes to mobile applications there is one key area that is most important to consider: the battery life.

It doesn’t matter what type of application you are developing, it is important to measure how the application impacts battery life. Any user will become irritated if the battery is drained too quickly, and for some types of applications, it can be absolutely crucial to consider performance if it is to be well received by the intended users.

An example could be an application that uses the GPS functions. It is likely that a person who uses such kind of application frequently will have the ability to charge the phone. If the battery is drained quickly, you can be quite sure that the user will be reluctant to use the application too much, if at all. As a tester, you can test this in a simple (but easily forgotten) way: ensure that you do not have your mobile device connected to a power source when you run your tests.

Security

Here it is important to analyze what types of potentially sensitive data the application manages. If it will handle sensitive data, such as personal data, bank accounts and the like, what happens when a session stops unexpectedly? Is the data deleted, or is there something left in the cache?

Can a user save the application on a removable memory card? What if you then move over the memory card to another device?

Is sensitive information stored in the phone encrypted or can it be accessed by people who have direct access to the pad / phone? These types of devices are unfortunately several times more likely to be stolen or lost, than a laptop or computer, making security issues of this type increasingly important.

Guidelines for operating systems

Finally, Apple, Microsoft and Google all have guidelines for the development of applications on their platforms. Ensure that early in the project, you read the guidelines of each operating system and see which requirements need to be met and tested for, and be sure to take advantage of the already example test cases provided! Note that even for HTML5 applications there are valuable requirements and guidelines available, even though you may not plan to release your application via one of the available stores.

Conclusion

As you can see there are a lot of new things to think about. A good testing strategy for mobile applications will be an excellent start to deal with the new challenges they raise. We hope you’ve found some inspiration and tips in this article that can make your work easier.

To sum up:

Choose which devices and operating system versions you want to focus on, you can not test everything.

Keep your eyes and ears open for news about future phones and operating system to avoid surprises.

Take advantage of mobile testing networks. Services are constantly evolving and may soon be one that suits you show up.

Create a mobile testing strategy/plan.

Unplug the power cord and run the tests in different environments and usage conditions.

Install the application on your private telephone, this makes it easier to take the role of an ordinary user.

Authors

Rebecca Holm is a consultant in testing and requirements management since 2007 and has held positions in industries such as retail, gaming, finance and energy. Rebecca has had roles including technical tester, requirements analysts and test manager in both traditional development and mobile development. She is also responsible for Konsultbolag1’s competence network within UX.

Hans Joelsson has 8 years of experience in testing and test management and has worked in IT security, online gambling, government and banking. Hans has a strong interest in mobile applications and operates Konsultbolag1’s competence network in the mobile area.