Solving the 3 Biggest Questions in Continuous Testing
Download as PDF, PPTX0 likes241 views
The document discusses continuous testing in the context of DevOps, defining it as the automation of tests within the software delivery pipeline to achieve rapid feedback. It addresses critical questions on the distribution of testing responsibilities across different roles, the appropriate tools for each persona, and the benefits of correctly implemented continuous testing. The key pillars include stable automation, increased coverage, and the integration of various tools and practices to support effective DevOps processes.
More Related Content
Similar to Solving the 3 Biggest Questions in Continuous Testing (20)
Solving the 3 Biggest Questions in Continuous Testing
- 1. 1 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Solving the 3 Biggest Questions in Continuous Testing
- 2. 2 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io 2© 2018, Perfecto Mobile Ltd. All Rights Reserved. Housekeeping 1 2 Ask us questions in the Q&A panel Join our live poll question 3 We will send you the recording and the slides
- 3. 3 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Today’s Speaker: Eran Kinsbruner • Chief Evangelist and author at Perfecto • Blogger and speaker • continuoustesting.blog • infoworld.com/author/Eran-Kinsbruner/ • enterprisersproject.com/user/eran-kinsbruner • 19+ years in development & testing • Author of “The Digital Quality Handbook” and “Continuous Testing for DevOps Professionals” • Twitter: @ek121268
- 4. 4 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Today’s Agenda 1 2 3 The State and Definition of Continuous Testing How Should Testing Types Be Dispersed Among the Three Personas and Throughout the DevOps Pipeline? Which Tools Should Each of These Three Personas Use For the Creation and Execution of Tests? 4 What Are the Key Benefits to Continuous Testing When Mapped Correctly? 5 Q & A
- 5. 5 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io What is Continuous Testing? Continuous testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback as rapidly as possible.
- 6. 6 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Continuous Testing in DevOps Has Never Been More Complex Digital Transformation is Happening Now. How Does It Impact Your Business?
- 7. 7 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io What the DevOps Process Probably Looks Like Process Impact Organizational Impact 2-3 WEEKS 1-3 WEEKS ❌ Unstable ❌ Labor-intensive ❌ Cluttered ❌ Slow • Manual testing. • Unreliable and flaky executions. • Long time to analyze results. • Increases time to release. • Increases risk and reduces flexibility during the cycle. • Reduces innovation time versus bug fixes time. • QA isn’t part of the daily cycle since they have limited feedback to share. • Testing holds back innovation. ”End of Cycle” Testing
- 8. 8 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io CONTINUOUS What the DevOps Process Should Look Like
- 9. 9 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io The Path to Continuous Testing Continuous testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback as rapidly as possible. Removes the testing bottleneck. Significantly reduces risk with on-going quality visibility. Allows fixes DURING the sprint rather than AFTER it. Significantly reduces developers’ bug fixing time and increases innovation time. Turns testing into VALUE instead of LIABILITY B E N E F I T S
- 10. 10 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io The Path to Continuous Testing 1 2 3 4 5 Stable automation Daily cycle Increase coverage Reach 95% Continuous testing W H A T Y O U ’ L L G E T W H A T Y O U ’ L L N E E D • 99.9% availability lab • Evidence collection • Process integration • CI • Defect tracking • Execution control • Dashboarding • Create scripts • Maintain scripts • Understand what’s wrong • Skillset matched tool • False negative detection in reports • Run scripts daily • Get results in minutes • Valuable coverage • Meaningful daily feedback • Advance validations • Visual automation • Basic orchestration • Up-to-date lab • Scaled lab • Analysis grouping • Automate all that possible • Advanced orchestration • Elasticity • Sharding • Environment control • Scaled reporting • Nightly Continuously • Cloud execution • Very high scale D E V O P S F R I E N D L Y Z O N E
- 11. 11 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io 1. How should testing types be dispersed among the three personas and throughout the DevOps pipeline?
- 12. 12 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
- 13. 13 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io A Plethora of Tools Support the Entire Pipeline
- 14. 14 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Let’s Focus on Testing Testing Is Led By 3 Main Personas B U S I N E S S T E S T E R S D E T ( S O F T W A R E D E V E L O P E R I N T E S T I N G ) S O F T W A R E D E V E L O P E R Performs exploratory and structured manual testing Creates functional and nonfunctional code-based test automation scripts (via Selenium, Appium, etc.) Creates unit and build acceptance tests
- 15. 15 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Business Tester: Manual? Exploratory? BDD? Codeless? How Should Testing Types Be Dispersed Among the Three Personas and Through the DevOps Pipeline? P L A N C O D E B U I L D T E S T D E P L O Y R U N DevOps Continuous Feedback Test Platform Test FW Trigger Feedback Time 90% Emulator/Local Device 10% Cloud Device Unit API Espresso / XCUITest Exploratory Pre-commit Manually – On-demand Few Minutes 70% Emulator 30% Cloud Devices API Espresso / XCUITest Commit / Few # day Automatic < 5-10 Min 100% Cloud Devices Espresso/XCUITest Appium Nightly Automatic < 6 Hours 100% Real Devices (S) Espresso/XCUITest 24/7 Automatic < 5 Min SDET Functional/NF Developer Unit/BAT
- 16. 16 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io 2. Which tools should each of these three personas use for the creation and execution of tests?
- 17. 17 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io 2/28/2019 17© 2015, Perfecto Mobile Ltd. All Rights Reserved. BDD/Codeless Scripting Flaky and common test failures and complex manual test cases Interactive Tests UI/UX manual tests High Skilled (Dev/SDET) code-based scripting Developers and SDETs Business Testers Ownership Business Testers Ownership Balancing Test Creation for the Three Different Personas with the Right Tools
- 18. 18 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io 2/28/2019 18© 2015, Perfecto Mobile Ltd. All Rights Reserved. Interactive Tests UI/UX manual tests Developers and SDETs Business Testers Ownership Business Testers Ownership Balancing Test Creation for the Three Different Personas with the Right Tools
- 19. 19 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Test Authoring Workflow and Skillset Required Define manual test scenarios, BDD & user stories style scenarios Create scripts in Java/JS etc. from within IDEs (IntelliJ, Eclipse) Define Objects using Object Spy tools, DOM viewers etc. Page based test steps creation Insert Visual validations and Assertions Typically takes longer to develop, more complex (~6 Hours per test) Medium-High code development skills required Test Maintenance Test changes are required proactively, tests are more error-prone to changes in production, objects Tests are managed and maintained in an SCM tool (GIT, Perforce) Tests are recorded with no coding in most cases Codeless tools UI used for creation Objects are “learned” and generated on-the-fly Test scenarios are often less structured and more exploratory based (flow based) Time to author is shorter, can be ~1 hour per test Test reusability is easy Business tester, lower to no-coding skills is sufficient Tools support self-healing with auto test correction Local or cloud test versioning is used, no GIT integration SDET/Developers Business Testers (&Developers?) Mindset and Workflow Changes in Test Automation
- 20. 20 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Tools Maturity Highly mature, includes samples, best practices, documentation Integrations exists to ALM tools, Defect management etc. Testing Types and App Types Supported Functional, API, Load, others Mobile Native (Appium e.g.) and Desktop Web (Selenium e.g.) Emerging technology, less mature, no well- defined guidelines and practices (e.g. guide to shift from standard to ML?) Web is more mature than mobile codeless, basic integrations only Mostly functional (E2E) and basic API Most support web, mobile is lagging behind SDET/Developers Business Testers (&Developers?) Test Execution Configured environment using tools like TestNG data provider Execution done locally, via CI, cloud-based Execution management is built-in the codeless tools Execution done locally, via CI, cloud-based Mindset and Workflow Changes in Test Automation
- 21. 21 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io 1. Identify manual and/or flaky test steps 2. Mark these as “Codeless” to disconnect them from the code-based automation suite (avoid duplications) 3. Record the tests in a “SMART” manner: - Flow based & Exploratory recording - Reusable components - Robust Object Locators - Tagging for reporting and tracking 4. Connect codeless to CI and pipeline for continuous testing 5. Identify maintenance and examination points for both suites 5 Recommended Practices for Codeless Testing
- 22. 22 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io 3. What Are the Key Benefits to Continuous Testing When Mapped Correctly?
- 23. 23 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io CREATION EXECUTION LAB ANALYSIS SCALE • Few executions/day • 2 platforms • 10 executions/day • 2 platforms • 1000 executions/day • 10-20 platforms • 25 concurrent • 10K executions/day • 25-50 platforms • 25-150 concurrent • 100K executions/day • 25-50 platforms • 1000 concurrent 1. Stable automation 2. Daily cycle 3. Increase coverage 4. Reach 95% 5. Continuous testing Q A D E V What Are the Key Benefits to Continuous Testing When Mapped Correctly?
- 24. 24 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io The Key Pillars of Continuous Testing in DevOps ANALYSISCREATION EXECUTION LAB Unified solution Serves DevOps with greater effectiveness. Lab for continuous testing that serves as the backbone for all testing activities. • Hosting of devices, management & logistics • Availability & scaling • Supporting functions Test execution within the pipeline • Scale • Orchestration • Frameworks Test creation and maintenance • Authoring tool • Validations Test analysis • Fast feedback • Root cause analysis
- 25. 25 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Perfecto’s Smart Automation for Continuous Testing SMART Test Execution Fast and parallel test execution with multi-team orchestration abilities and management. SMART ML-Driven Test Reporting and Analysis "Single pane of glass" provides visibility and scales to support millions of test results. SMART Test Creation Automated creation that matches your team’s skillset (Appium, Espresso, XCUITest, Quantum BDD). SMART CLOUD Lab Always on and stable. Always up to date. Supports all mobile OS and platforms. The Perfecto human factor increases your chances to succeed. Black Belt Testing Experts Training Dedicated Success Manager 24/7 VIP Support
- 26. 26 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io About Perfecto
- 27. 27 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Perfecto Is a Perforce Company A G I L E P L A N N I N G C O D E M G M T. & C O L L A B O R AT I O N E M B E D D E D C O M P O N E N T S O P E R AT I O N S T O O L S A U T O M AT E D T E S T I N G
- 28. 28 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
- 29. 29 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Thank You