Mobile Device Client Application Performance Testing

Mobile Device Client Application Performance Testing

• 1.
  Mobile Performance Testing Part1: How to Performance Test Mobile Client Applications on a Mobile Device
• 2.
  Series of 3Mobile Performance Webinars Part 1: Clients applications – Now Part 2: The Server – Feb 21 Part 3: The Network – March 14 All at 10 AM EST
• 3.
  Alan Trefzger Newman Yang Jan Princen
• 4.
  Software Quality Improvement •XBOSoft – Software QA Consulting and Testing – Currently 100+ QA experts – San Francisco, Beijing, Oslo, Amsterdam
• 5.
  Quality Life Cycle
• 6.
  “Proven Results” “Our products are very complex. We expected a one year ramp up time, but XBO was able to contribute to our release cycles in just 6 months." - Development Vice President, Hyperion-Oracle "The results are very encouraging. Thanks a lot for all the effort put into this project. We really appreciate your work." - Manager, Quality Assurance, Autodesk "Everything you've done is impressive and we are quite pleased, well done! So keep up the good work." - QA Director, CA
• 7.
  Agenda • Importance ofMobile Performance Testing • Types of Mobile Performance Testing • Mobile Performance Testing for Local (Android) Applications – Setting up a performance test – Using different tools to test • Case study – How fast can Angry Birds run • Evaluate the tools • Q&A
• 8.
  Importance of Performance •Mobile internet traffic expected to pass desktop internet traffic in 2014. • By 2015, over half of all mobile subscribers are expected to be engaged in m-payments . • Amazon reported that the company calculated it lost 1% of sales for every extra tenth of a second required to load a page. • Google experiment: traffic and revenue fell by 20% when the pages with more results took an extra half second to load. • Akamai found that the number of users who abandon a page after 3 seconds was 57%. - Mobile Site Optimization, Sep 2011, Strangeloop
• 9.
  Types of MobilePerformance 1. Client Application / device performance 2. Server performance 3. Network performance
• 10.
  Server Performance Following aresome of the challenges, variables, and issues covered in the Server Performance webinar. • Amount of data sent, • Number of communications between Server and Client/Browser • Tools • Optimization Strategies
• 11.
  Network Performance Following aresome of the challenges or variables covered in the Network Performance webinar. • Latency, jitter, packet loss,… • 2.5G, 3G, 4G LTE • CDNs
• 12.
  Local Device andApp Performance Why focus on the local device and App? • Discover performance based on different hardware/software configurations with the same application for a defined task. • Discover which platform meets a set performance objective with the least cost. • Determine which platform uses the least resources.
• 13.
  Performance is importantto almost every application or task Buy a product Obtain a bank balance Look up a ticker symbol Kill a man with Download one page spear (WoW) of information Start an application (game)
• 14.
  How Fast canAngry Birds Run?
• 15.
  Performance Test Strategy Performancetest on a few devices, and extrapolate that it will work on those devices that are more powerful (more memory, higher CPU Hz, etc…, 5 criteria are defined later). Different than functional test. It may work functionally on a certain device, but not meet performance requirements. Example, for Android, you can test on a well chosen set of 7-15 devices and be confident that your application will work functionally on the majority of Android devices. However some of those devices may not meet performance requirements.
• 16.
  Defining the test ProductManager wants to find the lowest phone that satisfies the performance requirements. • Requirements: – Launch Angry Birds in 15 seconds – Throw an Angry Bird far away in 8 seconds – Play Angry Birds and play music at the same time – Play Angry Birds and at the same time delete 100 SMS’s in 5 seconds Lowest solution that fits requirements Using tools and manual testing to make purchasing decision
• 17.
  Steps for performancetest 1. Define task, requirements and test. 2. Choose a sample set of devices. 3. Choose one or more tools and run them. 4. Compare to manual test. 5. Use Quadrants’ DB of devices to eliminate those phones which do not meet performance requirements. 6. Chose a mobile device based on criteria other than performance.
• 18.
  Configuration of devices •3 popular Android devices
• 19.
  Tools • Quadrant –measures CPU, Memory, I/O, 2D and 3D. • Linpack - measures CPU floating point performance by single thread and multi thread. • Smartbench – Simple tool to get idea if the device is suited for example gaming. • Others: Jbenchmark, SPMark, Nenamark
• 20.
  Quadrant Result Total Scoreaverages the scores of the 5 specific criteria 3500 3000 2500 2000 Lenovo A1 1500 HTC Desire HD 1000 Motorola 500 ME860 0 Total Score CPU Memory I/O 2D 3D
• 21.
  Quadrant Mobile DB
• 22.
  Using Linpack 60 50 40 Lenovo A1 30 HTC Desire HD 20 Motorola ME 860 10 0 Single Thread Multi Thread MFLOPS - Millions of Floating Point Operations per Second Lenovo A1 HTC Desire HD Motorola ME 860 Single Thread 9.91 39.01 38.94 (MFLOPS) Multi Thread 8.55 32.51 51.96 (MFLOPS)
• 23.
  Using Smartbench 3000 2500 2000 Lenovo A1 1500 HTC Desire HD 1000 Motorola ME860 500 0 Productivity Game Motorola ME860 >HTC Desire HD > Lenovo A1
• 24.
  Manual Test AngryBird
• 25.
  Manual Test Result- Angry Birds Scenarios Required Lenovo A1 HTC Desire HD Motorola ME 860 1. Launch Angry Bird 15 seconds 19 seconds 14 seconds 10 seconds V2.1.1 2. Throw the bird to a 8 seconds 8 seconds 6 seconds 5 seconds far distance 3. Play Angry Bird Slight V2.1.1 and play music at Continuous Continuous Continuous discontinuity the same time 4. Play Angry Birds and delete 100 SMS 5 seconds 4 seconds 3 seconds 2 seconds simultaneously
• 26.
  Results of Manualtest • Lenovo A1 - didn’t meet requirements • HTC Desire HD - met requirements • Motorola ME 860 - met requirements Conclusion • Since the Lenovo A1 device didn’t meet the performance requirements, any device with a lesser Quadrant score than the Lenovo will be rejected. • Any device with an higher score than the HTC device will be accepted.
• 27.
  Tying it alltogether 1. Define task, requirements and test – Angry Birds, certain speed, multi apps 2. Choose a sample set of devices – Lenovo, HTC, Motorola 3. Choose a tool (0ne or more) – Quadrant, (Linpack, Smartbench) 4. Compare to manual test – Find out which device conform to performance requirements 5. Use Quadrants’ DB of devices to eliminate those phones which do not meet performance requirements. 6. Chose a mobile device based on criteria other than performance.
• 28.
  Thanks Questions and Answers www.xbosoft.com [email protected] 408-350-0508 28