How to Prepare Your Apps for iOS 14 - Test Strategy, Coverage, & Best Practices

Perfecto by Perforce © 2020 Perforce Software, Inc.
How to Prepare Your Apps for iOS 14:
Test Strategy, Coverage, & Best Practices

2 | Advanced Codeless Testing for Web Apps
Today’s Speakers:
Eran Kinsbruner
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 20+ years in development & testing
• Author of “The Digital Quality Handbook”
and “Continuous Testing for DevOps
Professionals”
• @ek121268
• https://www.linkedin.com/in/eran-kinsbruner-4b47a81/
Anand Bagmar
• Software Quality Evangelist at Applitools, blogger, and speaker
• Over 20+ years in software development and testing
• Specializes in product quality strategy & execution, and building
automated testing tools, infrastructure, and frameworks
• Built open source tools related to software testing: WAAT (Web Analytics
Automation Testing Framework), TaaS (for automating the integration
testing in disparate systems) and TTA (Test Trend Analyzer)
• https://in.linkedin.com/in/anandbagmar
• @BagmarAnand

perfecto.io3 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Today’s Agenda
1
3
4
Mobile Test Coverage Guidelines
iOS 14 Features Overview and Pitfalls
5
How to Combine Testing on iOS Simulators & Real Devices
Live Demo: iOS 13 vs. iOS 14 Issues (Appium)
6 Q&A
2 Functional & Visual Testing — Recommended Strategy

• New UI Enhancements
• Widgets
• App Library
• Messages
• Notifications
• New translation app
• Enhanced Siri
• New Safari browser (Safari 14)
• Car Key/Car Play
What’s New in iOS 14

What’s New in iOS 14 — Continued
• App Clips load within seconds and let users
complete specific tasks, like pay for parking using
Sign In With Apple and Apple Pay. App Clips can be
discovered and accessed by scanning Apple's new
"App Clip codes," or by using NFC and QR codes.
They can also be shared in Messages and Safari.

iOS 14 Collected Pitfalls: Functional, Memory, UI Issues
Works on iOS 13,
Crashes on iOS 14
Works on iOS 13,
Crashes on iOS 14

iOS 14 Collected Pitfalls: Functional, Memory, UI Issues
Looks Good on iOS 13 Text Field Truncated on iOS 14

• Quality Strategy = Testing + Automation + App Release Plan
Quality Strategy

Quality Strategy

• Build in quality by testing early
• Feature kick-off
• Business requirements
• Story kick-off
• User impact
• Dev-Box testing
• Implement what was expected
• Thorough story testing & exploratory testing
• Test was was implemented
• Product owner sign-off per story & feature
• Involve everyone, quality is a team responsibility
Structured Way of Working

• Automate as much as you can (what brings relevant value)
• Unit tests
• Integration tests
• Functional tests (E2E) — focused on user-journeys
• Visual testing — user experience should be consistent, and as expected
• Analytics tests
Automation Is Key

Value of Automation
Run the tests on every change — capture the
regression as soon as it is introduced in the code.

Simulate real-user behavior/interactions.
Run the tests on simulators & real devices.

• In-house testing
• On-field testing (sanity)
• Staged releases
• Sentiment analysis
• Review comments (PlayStore/App Store)
• Social media (Twitter, Instagram, Facebook, etc.)
• Extensive monitoring of staged releases
• Analytics
• Crashes
• Support tickets
Release Approach

Does your strategy change if a new version
of OS needs to be supported?

Eran gave plenty of examples why it is necessary to keep the variables in mind!
YES!

• Validate compatibility of your product with the new iOS version, and quickly!
Key aspects to keep in mind in case of iOS 14.

• Functionality/gestures
• UI/UX
• Memory
• New installations vs. existing installation vs. upgrade app versions
• Integrations/permissions
• Caching/data storage
Challenges You May Face

• Executing automated tests
• Unit tests
• Integration tests
• Functional tests (E2E) — focused on user-journeys
• Visual testing — user experience should be consistent and as expected
• Exploratory testing — finding impact of iOS version change on your app
Strategy for iOS Version Change

Test Automation Coverage Guidelines

Balancing Test Automation Coverage
TEST SCENARIO ONE
Test Case – 1 Test Case – 2 Test Case – 3 Test Case – N
2 sides of the same coin

Test Coverage Fundamentals
Web Traffic Analytics
Selecting the Right Test Cases

iOS and iPadOS Testing Coverage

• iOS 13 (Latest–1) testing required on
• iPhone 11 series
• iPhone 6 and below
• iPad Air/Pro Series
• iOS 12.4.8 (Latest –2)
• iPhone 6
• iPad Mini 2
iOS and iPadOS Testing Coverage: GA, Latest-1, Latest-2

Latest GA Versions:
• Chrome 84
• Firefox 79
• Edge 84
• Safari 13
• Safari 14 Beta*
Global Browser Market Share Update: Adoption Trends Vary

Mobile Testing & Desktop Web Testing Are One
iOS Safari
Android Chrome
Desktop Web UI

Balancing Virtual Devices and Real Devices

Virtual Platform Definitions
• Simulators
• iOS (Apple)
• A simulator is software that imitates the workings of
software on your system that was made for mobile.
• Mobile simulators can be used in app testing and cross-
browser or responsive testing to simulate a mobile device
and running your app or website in it.
• Emulators
• Android (Google)
• An emulator is software that imitates the working of
software on your system that was made for mobile.
• Emulators also take into consideration the hardware part
of the mobile device (memory, processor, battery
performance, etc.).
Real Device Behavior

Differences Between Android & iOS Virtual Platforms

Considerations
for Using iOS and
Android Virtual
Platforms vs. Real
Devices
Stage and activity in
the software
development cycle
Dev
Test
Debug
Design
Cost avoidance
Coverage
Platforms
Real user
conditions
Test types
Functional/Regression
/Integration Testing
Unit
Non Functional
Production Monitoring

Perfecto’s Mobile Testing Pyramid Combo
Testing Pyramid Within Your DevOps Pipeline Requirements
U N I T T E S T S
M A N UA L
T E S T S
Testing isolated units of code.
I N T EG R AT I O N T E S T S
Testing across integrations of more than one system.
U I T E S T S
Testing through a user interface.
Per code commit/ Post commit
Build Acceptance/Regression Testing
E2E Testing
Exploratory/Manual Testing
Dev
Triggered
Mix Real Devices & Emu./Simu.
CI
Triggered+
Real Devices Only
CI
Triggered
Real Devices OnlyProd. Monitoring
Non Functional Testing:
Performance (UX)
Testing/Security
Real Devices Only

Virtual Platforms
Mix of
Virtual Platforms & Real Devices
Only Real Devices
Pipeline Testing Coverage Example

Demos: Cross Platform Testing With Perfecto and Applitools

Continuous Testing – Key Metrics/Categories

Perfecto’s 4 Key Pillars of Continuous Testing in DevOps
This is where money is spent (Time, Tools, Resources)
This is where value is being
realized, quality is improved
Unified solution serves DevOps with greater effectiveness.

We Invented Visual AI that replicates the human eye and brain, looks at every screen
and page in seconds, and spots app regressions that truly matter.
We Help App Teams with their:
• Functional Testing & Visual Testing
• Web, Mobile, and UI/UX Testing
• Cross Browser & Device Testing
• Content Migration/OS version upgrades/browser version upgrades
• Compliance Testing, Localization, Accessibility, PDF, & More
Applitools Visual AI has analyzed one billion+ images with 99.9999% accuracy.
About Applitools

Visual AI
Industry’s only AI powered computer vision
replicates the human eye & brain.
SDKs & Integrations
SDKs to augment all modern test frameworks,
and integrations with popular source control,
CI, and defect tracking systems.
Perfection Across All Screens
Ultrafast Grid ensures visual perfection across
all browsers, screens, and viewports.
Ultrafast Grid Architecture >>
Speed of CI/CD
Execute tests in seconds vs minutes for faster
builds and on-time delivery.
Ultimate Efficiency
Create tests faster & with fewer brittle locators
and labels.
Increase Test Coverage
Open-ended, AI-powered assertions reduce
code and maintenance while increasing test
coverage.
Applitools Cloud
Visual AI Platform
One Intelligent Platform
Adding Applitools Visual AI to your existing test automation framework allows quality teams to release
perfect apps faster and at a reduced cost.

Thank You!
perfecto.io or follow us @perfectomobile
applitools.com / @applitools

How to Prepare Your Apps for iOS 14 - Test Strategy, Coverage, & Best Practices

More Related Content

What's hot(16)

Similar to How to Prepare Your Apps for iOS 14 - Test Strategy, Coverage, & Best Practices(20)

More from Perfecto by Perforce(16)

Recently uploaded(20)

How to Prepare Your Apps for iOS 14 - Test Strategy, Coverage, & Best Practices