Mobile App Testing by Mark Wilson

© Planit Software Testing
Mobile App Testing & Other
Considerations
Presenter: Mark Wilson

Our knowledge is your power
Software Testing
Consulting, Advisory, Delivery,
Managed Services, Training

Why is Mobile Computing Important?
People worldwide now spend more time on their smartphones than on traditional PCs.
Smartphone sales surpassed PC sales in 2011.

64%
15 - 65
The Mobile Landscape In New Zealand
Have a smartphone now

90%
15 - 65
Will have a smartphone
by 2018

44%
2013
Have a tablet at home now

78%
2018
Will have a tablet by 2018

Apps vs Web

The Future Of Mobile Apps

What Is Mobile Computing Used For?
• Apps continue to grow
• Gaming and Social Media are over 60% of apps usage

How Successful Are most Mobile Apps?
Apps on our smart phones
• Average 25-26 apps
• 1 in 4 apps downloaded are never used
• On average 22% of Apple users who download an app
only use it once.
• On average only 34% of Apple users who download an
app will use it more than 11 times.
• Over 1 million applications are available from the app
store alone

What Spurs Mobile App Growth?
Mobile app usage worldwide.
What will be the next big thing?

Do Mobile Apps Have Problems?
Have you had problems with a mobile app within the
last 6 months?
NO
44%
YES
56%

Types Of Problems Encountered?
•The mobile app crashed/froze/
displayed an error
•The mobile app was slow to launch
•The mobile app would not launch
•The mobile app didn’t function as expected
•None of these

What The Customer Expects
•Things to work first time
•They expect things to work fast
•They expect things to be intuitive
•They expect instructions to be clear and self explanatory
•Your app generally has to be usable by people with differing
abilities
•Most apps don’t come with a 1-2 day training course! It’s a
Web App not SAP

So Why Do Mobile Apps Fail??
Customers Can’t Make It Work!!
1. They have problems installing
2. They don’t run properly
3. They freeze or lock

The App Won’t Install, Won’t Work Or Freezes!
Why is this? 3 Answers:
• They have not been functionally tested properly!
• They have not been useability tested on that platform combination.
• They have bugs which were not found in testing.
• This application documentation clearly states that this device in this
configuration is not supported.
People Love to
Complain

Scope Targeted Platforms
What platforms will you target and test your
application on?
• Operating Systems?
• Browsers ?
• Hardware?

Scope Targeted Platforms?
Each one has multiple versions of the
operating system

Targeted Platforms?

Targeted Platforms?
What Devices?
Phones – Iphone 3-6, Galaxy S3-S5, Nexus 4-7
Blackberry ??, HTC ?, Nokia windows phone,
Huawei, Redme + hundreds of clones etc.
Tablets – IPad 1-4, Galaxy Note 1-4, Windows
Surface ??. Le-Novo, Sony, Acer, Asus, Amazon
Kindle, HP, IPOD Touch + Hundreds of clones.
Different screen sizes
Different resolutions
Different memory
Different processing power
Different operating systems
Different quality antennas

153
Targeted Platforms?

Lessons Learnt – Devices
Manage your platform scope
• How many devices, operating systems, versions?
o iOS?
o Android?
o Windows Phone?
o Blackberry, Symbian, . . .?
o Smart Phones, Tablets, . . . ?
• Your systems future?
o Device & OS fragmentation
Requirements

Problems with Look Feel and Function
1. The application doesn’t do what the customer thinks it should
2. The application is just too complicated or confusing
3. It doesn’t look very good or is isn’t nice to use (Subjective but
unfortunately customers are)

The Application Is Not Intuitive
• Make your app as clear and easy to use as possible.
• Keep a common look and feel throughout the application.
• Get your kids to trial it, then get you mum and gran to have a play.
• There are guides around usability – look at them. Government have
usability and accessibility guidelines which are being enforced more
rigorously.
• If you are targeting a bigger audience then look at the issues around
accessibility. http://blindfoundation.org.nz/learn/information-for-
businesses/accessible-websites is a good place to start.

Lessons Learnt – Look/Feel/Function
Have a clear vision for the application
• Put together clearly defined requirements (‘Just like our website’ is
not good enough!!)
• Target your market demographic
• Look at what is successful in that market
• Look and feel can be as important as function
• But look and feel won’t keep your customers happy if the
application doesn’t do what it’s supposed to
Requirements Requirements

Be Prepared To Speed Up
First to market is a big advantage!

Focus on time – everything moves quicker on mobile projects
• The market is moving incredibly fast
• The technology is moving very rapidly
• The fashions and trends are moving fast
The Application Takes Too Long To Get To Market
http://williesun.com/tag/ramble-
away/?doing_wp_cron=1402289455.7205860614776611328125
Remember you are better off being
second to market with a product that
works than being first with one that
doesn’t

A Newer, Quicker Way Of Working!
• Manage your stakeholders
• Get marketing and the business involved early
• Gear up for more releases

Focus on time
Look at continuous integration
Automated functional regression testing
Automated regression performance testing
Automated penetration testing/vulnerability scanning
Service virtualisation to allow parallel development
Farm out the bulk of your device usability testing
Look To Reduce Cycle Times – Not Quality!

Prepare For Regular Release Cycles
• Plan for your ongoing release cycles
o Business will see new opportunities
o Expect Public APIs to change (e.g. Google Maps, TradeMe)
o Technologies will change
o Taste and fashion change
• How will you deploy future releases?
o Plan for availability of infrastructure

Lessons Learnt – Be Prepared To Speed Up
• Be prepared for small projects
o With frequent follow-ons
• Build your team early
o Get the right mobile expertise in at the start
• Connect early to the business stakeholders
• Get your scope nailed early
o Especially what devices and OSs you will support
• Start small
o Plan for measured rollouts
• Plan your device management and your security early
Requirements Requirements Requirements!

Test Improvement
Consider:
• Testing smarter
• Device emulation tools
• Test automation

Test Smarter
Consider
• Do you need to run full functional regression tests against
every device combination?
• Could you use Bluetooth keyboards to reduce fatigue?
• Could you use an adaptor and a full size screen?

Device Emulation
Pros:
• Makes many device combinations available quickly and affordably
• Testers use a proper desktop to drive the application
• Defect creation is faster using cut and paste, screen shots etc.
Cons:
• It’s a step away from production like. You have introduced another
layer that can create and/or mask problems.

Test Automation
Consider
• You may need test automation to enable you to perform
regression testing on each release in a timely manner
• Do an ROI calculation before you invest in this area
• If you are going to invest in this area do it early

Lessons Learnt – Test Improvement
Emulation can speed up testing but is not a substitute for
testing on the actual device.
Look in to test automation early.
Devices:
• Track carefully – they can go walkies . . .
• Testing infrastructure – rent or buy, Planit Mobile Test Lab
• Bluetooth keyboards can save the thumb
• Screen adaptors can save eyes
• Invest in antibacterial wipes

Users Expectation
Mobile Apps vs Mobile Web
Performance is important in the
mobile world

Users Expectation
So what are your customers
expectations?

Users Performance Expectation

User Expectations For Mobile App Performance

The Apps Are Too Slow!
• Performance is a really big deal.
• How do you test mobile performance?
• What is good performance?
• What is the total impact on the performance of your
system?

We are not fully in control
• The users location determines the strength of signal they have at the
time
• The network provider and plan they use affects performance
• The base performance of their device can vary
• There could be other applications running on their device at the same
time
As a general rule – we cannot control these!

Mobile
Networks
System Under Test
F
5
F
5
APP
APP
APP
APP
Legacy 1
Address
FWK
Account
SystemInternet

System Under Test
F
5
F
5
APP
APP
APP
APP
Legacy 1
Address
FWK
Account
System
x 220
Staff
x 10
Manager
x 20
Accounts
Mobile
Networks
Internet

To get production like response times you need:
• A production like environment to test against
• To create a production like load on all components
• A production like test data selection
If it isn’t production like then you introduce the risk of missing
a problem or creating a problem! Also, the response times you
see during testing most likely wont be representative of
production.

You can’t control what people have running on their devices, where they
use it or who their provider is!
So keep it mean and lean!

What is good performance?
1 second response time?
2 second response time?
Whatever enables your
customers and staff to work
efficiently without getting
frustrated.

Lessons Learnt – Performance
• Think about performance testing early
• Consider the whole system not just the mobile component
• As production like as possible:
o Environment, hardware, configuration, application versions
o Data populated to production like levels, test cases selected in a production like
way
o Production like transaction mix and durations
o Production like load on all system components.
• Look to test at lower levels earlier e.g. web services.
• Look to introduce regression performance testing if appropriate.
• Be aware of the performance of any third party services such as payment
gateways and try to ensure you can see the performance costs in your logs

Security Issues?
90% of top mobile banking apps
have security issues
(IOActive, Jan 2014)
90% of mobile apps have at least 1
vulnerability
(ZDNet, Nov 2013)

Security Security Security
The top 10 security issues faced by Mobile devices:
1. Lack of passwords
2. Lack of 2 factor authentication
3. Wireless transmissions not being encoded
4. Mobile devices are susceptible to malware
5. Mobile devices often don’t have security software
6. OS is out of date and doesn’t contain latest security patches
7. App software is often out of date
8. Unsecure ports on devices can be open
9. Mobile devices can be modified, jail broken, or rooted which can allow unauthorised access
10. Unsecure Wi-Fi can allow access to the device.
Consider BYOD

Lessons Learnt - Security
• Don’t forget mobile security
• Address BYOD
o Strategy to manage apps
on devices
o Security policies
o User agreements
www. webhostinggeeks.com

Mobile Projects Growing in NZ
• Telcos
• Banks
• Insurance
• Media
• Government
• Entertainment

How Our Projects Are Changing
Business changes lead to new projects
• Mobile projects are increasing
o In quantity
o In business importance
o And in speed
– “First-mover advantage”

Summary
• Growing business pressures on schedule
• The user / customer experience is critical
• Mobile projects are increasing
o In quantity, in business importance, and in speed
• Lessons Learnt
o How many devices, versions?
o Define requirements
o Involve the business early
o Involve testing early
o Device management

Mark Wilson
Principal Consultant
mwilson@planittesting.co.nz
+64 27 367 1939
Lessons Learnt In Managing And Delivering
Mobile Technology Projects
Contact Planit
www. planittesting.co.nz
+64 4 815 8199

Mobile App Testing by Mark Wilson

More Related Content

What's hot (20)

Viewers also liked (14)

Similar to Mobile App Testing by Mark Wilson (20)

Recently uploaded (20)

Mobile App Testing by Mark Wilson