Mutation Testing

Mutation Testing

• 1.
  TejaSoft Innovations Pvt.Ltd. [email protected], +91-9886723872
• 3.
  UNIT TESTS QUALITY BENDING #techdebt Byrecognizing Fake Tests
• 4.
  TYPES OF TESTING
• 6.
  PURPOSE OF UNITTESTS To know what developers have understood as method level requirment Avoid repeat manual pre-commit testing by Developers To make sure regressions are caught if the main code changed To make sure Developers get to know bugs much much before QA team can report Bring in Automation to speed up development Help Developers Identify the Unit Level Mistake (reduce debugging time) & Fix bugs with ease
• 7.
  UNIT TESTING SMELLS/QUALITYGAPS • Empty Unit Tests • No Asserts • Wrong Asserts – They should have failed but pass • @Ignore and commented test code Fake Unit Tests • Even after 100% Cover Coverage, still many production or QA reported issues • New Code that conflicted past requirement should have failed past Unit tests, but it does not Tests are Not Enough • Too Much of Mocking • Flickering • Interdependencies • Long Running • Magic Numbers Complexity
• 10.
  Vaccine created DEFENCEMECHANISM SHOULD KILL (Unit Tests should fail) mutated VIRUSES – then Vaccine is effective If Mutated VIRUS survives (all unit tests pass), then Unit Tests are not enough.
• 11.
  WHAT DOES MUTATIONMEAN? Mutation - Change Main CODE and see if Unit Tests catch (fail) What is mutation testing? Bugs, or mutants, are automatically inserted into your production code. Your tests are run for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. The higher the percentage of mutants killed, the more effective your tests are.
• 12.
  HOW TO CREATETHE MUTANTS! Operator mutations About 30 different patterns like replacing + by - , * by ** and similar, but also > by >= Keyword mutations Replacing true by false , in by not in and similar Number mutations You can write things like 0b100 which is the same as 4, 0o100, which is 64, 0x100 which is 256, .12 which is 0.12 and similar. The number mutations try to capture mistakes in this area. mutmut simply adds 1 to the number. Name mutations The name mutations capture copy vs deepcopy and "" vs None . Argument mutations Replaces keyword arguments one by one from dict(a=b) to dict(aXXX=b) or_test and and_test and ↔ or String mutation Adding XX to the string Collections mutations Add Junk Elements or Remove Few elements from List or Array Value Modify Modify the VALUE of results of method calls Inject method calls Add get or setter methods multiple times in place of one call Comment the CODE Remove some code or stop the methods being called https://pitest.org/quickstart/mutators/
• 13.
  3 TYPES OFMUTATION TESTING 1. DECISION MUTATION control statements are to be changed 2. STATEMENT MUTATION developer cut and pastes a part of a code of which the outcome may be a removal of some lines 3. VALUE MUTATION values of primary parameters are modified
• 14.
  WARNING May take FOREVERTO RUN
• 15.
  CODE - WORKSHOP
• 16.
  ‘KATA’ MODULE –CODE WALKTHROUGH 1. Company and Employee classes 2. CompanyRunner – Explain & Execute Main – With Code Coverage 3. CompanyTest – Explain & Execute Unit Tests – With Code Coverage 4. Run PITEST kata – Explain, Execute and Show the HTML Report
• 17.
  ‘FIX’ MODULE –CODE WALKTHROUGH 1. CompanyRunnerFixed – Explain & Execute Main – With Code Coverage 2. CompanyTestFixed – Explain & Execute Unit Tests – With Code Coverage 3. Run PITEST fix – Explain, Execute and Show the HTML Report 4. CompanyFixed and EmployeeFixed classes
• 18.
  CODE COVERAGE VSMUTATION Code coverage doesn't tell you everything about the effectiveness of your tests. Think about it, when was the last time you saw a test without an assertion, purely to increase the code coverage Imagine a sandwich covered with chocolate paste Code coverage would tell you the bread is 80% covered with paste, does not differentiate what kind of paste is it. Mutation testing, on the other hand, would tell you it is actually chocolate paste and not... well... something else if paste is not chocolate
• 19.
  REGRESSIONS VS MUTATION- TESTING 1. Regressions are unintended Since the vast majority of tests have been added due to previous bugs hence, if they find a problem, you have regressed to a previous state (in which the problem once again exists). High Failure Rates is, failure of the main code quality 2. Mutation are intended In Mutation testing framework would intentionally introduce small errors (called mutations) into your application and test failure are expected. More failures rate is high quality of Unit Testing
• 20.
  REFERENCES LANGUAGE Mutation TestingFrameworks Java 1. PIT (GitHub) https://github.com/vmzakharov/mutate-test-kata - Big Thanks to @vmzakharov, as code and insights of what all I shared is his CODE only (#tejasoft https://github.com/nagkumar/mutate-test-kata only refactored it) Different Mutators: https://pitest.org/quickstart/mutators/, https://pitest.org/quickstart/commandline/ 2. Jumble - http://jumble.sourceforge.net/ 3. µJava - http://cs.gmu.edu/offutt/mujava/ 4. Jester - http://jester.sourceforge.net/ 5. Javalanche - https://github.com/david-schuler/javalanche 6. tsDetect : https://github.com/TestSmells/TestSmellDetector - Code Smells Detector C / C++ Mull C# Visualmutator, Nester JavaScript Stryker PHP Infection (formerly called humbug) Ruby Mutant Rust mutagen Swift Muter Python https://pypi.org/project/mutmut/
• 21.
  MAINTAIN DISCIPLINE Do notcut corners, else end up with code and tests of “usual (IT Services)” quality
• 22.
  Q&A What Gives atmost Engineering Kick & Confidence? Production No Production Fires - Code Just Works Developers Unit/Mutation Tests Keep Failing QA Team Developers accept BUG as Bug Better code Each Day