Have you ever started using a new mobile app and got the feeling that no one tested the app to make sure it works? Or, maybe your customers started using an app that just went live but they can't get it to work.
That shouldn’t happen, says computer programmer and Inflectra founder Adam Sandman, who talked to Aviation Pros about the importance of mobile app testing and how it’s done.
Mobile app testing should be done to test that the applications work as they were intended to work, Sandman says, emphasizing that the end-users shouldn’t unwittingly find themselves being the app testers. If an app doesn’t work, people are going to be frustrated. They’re going to contact customer support, where employees are likely already trying to do more with less. Conversely, if employees can’t get an app that they need to work, both employees and customers may be annoyed.
According to Sandman, “the goal of testing an app is to make sure it works correctly according to the specifications, is usable and it also works on the devices that people need to use it on.”
“Usable,” he clarifies, means using the app should not require lots of instructions, manuals or handholding.
So what does mobile app testing look like? First, know that it doesn’t always look the same. Sandman outlines testing varies based on the type of application, the operating system and the device it’s used on.
The Device
When apps are created for aircraft maintenance providers or ground handers, the apps are most likely created for a specific type of device (for instance, Samsung or Apple) for a specific task, and the number of devices the app is used on is relatively small.
“When you talk about consumer apps (in an airport setting),” he says they need to be tested on any device that someone coming to the airport may use. Because it’s nearly impossible to test every single device that exists, he says, “You may have to make some assumptions about consumers using iPhone 9 to 13.” Then the app could advise, “This does not work with an iPhone 7,” for example.
The App “Flavor”
Before you do any testing, you have to understand “what flavors of apps” are you dealing with, Sandman says.
Is the app a web application running on a browser? Or is it from the app store (native apps)?
“The technology and tools to test a browser-based app, whether it's mobile or on a laptop or computer, are basically the same but completely different than testing a native app running on Android,” he says. “And it's very different from testing an iPhone (iOS) app. So there are three possible kinds of apps you're going to be looking at: web, Android or iOS.”
Types of Testing
Once the differences are determined, the question is how is the testing going to be done – by a human (manual testing) or a computer?
“The reality is, for almost every app, if you want to test the eventual usability, you do need to have some human testing, as in a person trying it out on a physical device, clicking the buttons, swiping, doing the gestures,” he says. “Aviation often requires tasks like checking in on a plane, a process that uses a QR code reader. So you're not just testing the app. You're testing the app and something else, like a passport or a driver's license or something that you're scanning to do something. Or you're scanning a bag tag. There's a lot of external stuff you have to test with the app.
“A human tester is going to try all the things you shouldn't do. Like, well, let me hit the keyboard and put 10 million characters into the login button. Let me try minimizing the window and click around. Or let me open up the login page a thousand times and do crazy stuff.
“Humans are really good at trying the crazy stuff. But the other thing with a mobile app, in particular, is you have these apps when you're mobile, you're physically doing something else (like walking or talking). And let's say it was an app that was supposed to be used in a vehicle, like a fuel truck, where it's gotta be hands-free. So you only get to really experience the app as a human when you're using it as if it would be used by the same person.”
Emulators or simulators are used to automatically test apps.
“They're basically virtual phones that run in a device cloud. Imagine a room where someone's brought every phone under the sun and put a giant bunch of servers and plugs all the phones in,” he says. And any kind of phone you ever want, any kind of tablet you ever want that exists is available in this giant cloud, this giant server farm. You, as a tester, can pay a fee every month and remotely access these devices. You will get what looks like a tiny graphic of a phone on your computer screen, and you can run your application on that virtual device.
“Now, instead of having to buy every iPhone model, you can test the app on an iPhone 7, iPhone 8, iPhone 10, Android or Samsung, and that means you can very quickly cycle through them all and try it all out.”
About 80% of the functionality can be tested virtually on a simulated set of devices, Sandman says.
“Once it’s determined that application works and you're sure it's pretty solid, you might then do the final 20% (usability) testing as a human on a physical device. Imagine you're walking down the corridor with a phone, with the app, trying to do the task. What you often find is what worked in a simulated lab environment – does technically work – but it's so unusable and the person is so tired, they won't bother. They'll just go call someone and say, where are my bags? And the whole point of the app was to reduce the cost of customer support in the airport."
Automated testing, having a computer simulate clicks and automatically running the test script can be done by an automation engineer.
If you're testing an app for refueling a plane, you need to have someone who's familiar with the task. "As long as a person knows the task, they can test the task," Sandman says. "But when you're talking about automated testing, you need someone who's more like a programmer, an engineer, who's writing automated scripts that will do that. And then what you find is that sometimes you need both people together.
"Automated testing will tell you if something works, but it doesn't tell if it's going to work well for the case at hand. Particularly with mobile and travel, these are some of the most important things."
How can testing be done efficiently?
Pairing a business user and a technical user together is probably the most efficient way of doing testing, he says. "You automate 80% of it and the remaining 20% would be manually doing it as if you were an end human, and that could be done by someone who's more of a business user.
"Because if it's an application where you've got someone doing automated testing, it requires someone who's more technically skilled in the technology, but they don't really understand how fueling a plane happens. So you pair a business user with a technical user. And one person understands how to do automated testing but has no idea of aviation. And the other person might be an aviation expert, but automation is something that they're not familiar with so you often want to pair them together."
When does a mobile app need to be tested?
Any time a major change is made to an app, it should be tested, Sandman says, "then any time there's a new major phone OS release, it should be tested as well."
If an app update is released every month, he says, "you should test it at least every month. If you release it every six months, you should check it at least every six months.
"Then, if there’s a major phone OS update, phone hardware update, or a new kind of handset, like a flip or a folding phone or new screen size comes out, you may need to check the app so it's going to be usable on that new device.
"Those are the three trigger points: app, phone OS and device.”
