12 functional-system-testing

12 functional-system-testing

• 1.
  Functional System TestingWritten by Adam Carmi
• 2.
  Outline Goal oftesting Test cases, test suites and test data What is functional system testing? Coverage Functional testing techniques: Functional analysis Equivalence partitioning Boundary value analysis
• 3.
  The goal ofsoftware testing The process of uncovering evidence of defects in software systems Does not include efforts associated with tracking down bugs and fixing them No amount of testing will improve the quality of a computer program The more testing we do of a system, the more convinced we might be of its correctness Testing cannot in general prove a system works 100% correctly
• 4.
  Test cases Thebasic component of testing is a Test Case In its most general form: ( inputs, expected-result) inputs include system state, user commands and data values to be processed expected result includes visible/audible interface changes or changes in the system state Test cases are organized into Test Suites functionality, security, performance, …
• 5.
  Test case executionA running of the software (under test) that provides the inputs specified in the test case and observing the results and comparing them to those specified by the test case If the actual result varies from the expected result, then a failure has been detected
• 6.
  Test data Aneffective test strategy requires careful acquisition and preparation of test data prior to testing Testing can suffer if test data is poor Test data concerns: Depth : quantity and size of data Breadth : variance of data values and data types Scope : completeness, relevance and accuracy of data Result of a query should be valid for the specific purpose of the query, and not due to a missing or inappropriate value Conditions : data should reflect specific “conditions” in the domain Data that would otherwise arrive after performing specific operations over time Test data and test results are expensive to construct
• 7.
  Example: Test datafor TVRS … … … … 8700342 longLastN longFirstN 000345 8245731 Josef Dina 255245 1346329 Porat Dan 264683 237812 Dan Levi 6386541 243567 Rachel Josef 8700342 Violation ID Offender’s first name Offender’s last name Issuing policeman ID Description: Violation records designed for validating violation lookup Name: test1.db
• 8.
  Specification Vs. ImplementationThe basic approaches to testing software are based on its specification and implementation White box testing – test cases and data are constructed based on the code that implements the software Quality and correctness of computations is validated Will not be further discussed in this tutorial Black box testing – test cases and data are constructed based solely on the software’s specification
• 9.
  Functional System TestingTesting of a completed application to determine that it provides all of the behaviors required of it Testing of completed increments that provide some degree of end-user functionality Search for defects that are variances between the actual operation of the system and the requirements for the system System is treated as a black box
• 10.
  How much testingis adequate? Completely validating IEEE 754 floating-point division requires 2 64 test-cases! float divide(float x, float y) From practical and economic perspectives, exhaustive testing is usually not possible Which software pieces should we test? Which test cases should we choose?
• 11.
  Coverage Coverage is a measure of how completely a test suite exercises the capabilities of a piece of software “ Each line of code should be executed at least once” “ One test case should be constructed from each specified requirement” It is necessary to use testing techniques that narrow down the number of test cases allowing the broadest testing coverage with the least effort
• 12.
  Technique: Functional AnalysisAnalyze the expected behavior of the system according to its functional specification Generate a test procedure for each of the possible usage scenarios Corresponds to use case scenarios Analyze how a change in one part of the system affects other parts “ Grand tour” test cases: the result of one test case produces the data that is the input to the next test case
• 13.
  Example: Functional AnalysisI Use Case: Remove Traffic Violation Supervisor calls for deletion of the chosen Traffic Violation TVRS prompts Supervisor for confirmation Supervisor confirms TVRS requests OffendersDB to delete the Traffic Violation from the offender’s record OffendersDB approves that the Traffic Violation has been deleted TVRS allows Supervisor to look up a new Traffic Violation as described in the “Lookup Traffic Violation” UC
• 14.
  Example: Functional AnalysisII Defect diagnosis: Actual results: Passed Failed Test results A message dialog stating that violation “243567” is not stored in TVRS Enter “243567” at “Violation ID” text field and press the “Search” button “ Lookup Violation” dialog is displayed Press the “Yes” button Confirmation dialog is displayed Press “Delete” button Expected result Action Related use cases: “Lookup Traffic Violation”, “Remove Traffic Violation” Pre-conditions: 1. TVRS initialized with test1.db database 2. Violation 243567 displayed in the “Lookup Violation” dialog Test case ID: 134543
• 15.
  Example: Functional AnalysisIII Verify effects of change Filled when the test case is executed How do we know that violation 243567 is stored in the system? Can a tester diagnose the cause of a defect? In addition, a query could be run on the Offenders database
• 16.
  Technique: Equivalence PartitioningIdentifies ranges of input and initial conditions that are expected to produce the same result A group of test cases form an equivalence class if: They test the same feature/scenario If one test reveals a fault, the other ones (probably) will too If a test does not reveal a fault, the other ones (probably) will not either It is adequate to use only a single representative of the equivalence class
• 17.
  Example: Equivalence PartitioningI Input value specification for “Lookup Violation” form: [a-zA-Z]{0, 10} Offender’s last name [a-zA-Z]{0, 10} Offender’s first name [0-9]{0, 9} Violation ID Valid values Field name
• 18.
  Example: Equivalence PartitioningII … … … … “” Empty [email_address] Invalid character Rachel Single known violation Dan Many known violations -1, 1234567890 ID < 0 or ID > 999999999 00243567 Known violation Violation ID 23ab@ Non numeric ID 32456720 Unknown violation “” Empty Offender’s first name Unknown violation David Character# > 10 Hasalongname Field Valid equivalent classes Valid representative values Invalid equivalent classes Invalid representative values
• 19.
  Example: Equivalence PartitioningIII The number of test cases to choose from is reduced to (3 + 2) × (4 + 2) × (4 + 2) = 180 The actual number can be further limited Single invalid field per test case (3 × 4 × 4 + 6 = 54) Importance of use case Resources available Most frequent input Life-critical software Infeasible test cases Randomly ...
• 20.
  Technique: Boundary ValueAnalysis Based on experience / heuristics Testing boundary conditions of equivalence classes is more effective Choose input boundary values as equivalence classes representatives Choose inputs that invoke output boundary values Examples: (0, 10] ⇒ validate using 0, 1, 2, 9, 10, 11 Read up to 5 elements ⇒ validate reading 0, 1, 4, 5, 6 elements
• 21.
  BVA as anequivalence partitioning extension Choose one (or more) arbitrary value(s) in each equivalence class Choose valid values exactly on lower and upper boundaries of equivalence class Choose invalid values immediately below and above each boundary (if applicable)
• 22.
  General purpose test-suiteconstruction technique May be used to obtain reasonable coverage with little effort Use cautiously! Unsuitable when values of different fields are related While test cases can be added Each new test case should include as many un-included valid non-boundary equivalence class representatives as possible While test cases can be added Each new test case should include as many un-included valid boundary equivalence class representatives as possible While test cases can be added Each new test case should include a single invalid equivalence class representative that has not been included before Manually replace/remove redundant or infeasible test-cases
• 23.
  Example: Country ClubI Specification 45 15 50 30 15 35 [60, 120] [16, 60) 50 25 45 70 25 65 [0, 16) 25 10 20 35 10 30 Age (years) Admission fee Guest status Visitor Member Student Visitor Member Student Day Sunday - Thursday Friday - Saturday
• 24.
  Example: Country ClubII A combo box is used for choosing the day and guest status 100, 60 , 120 [60, 120] Member Member Student Student Fri, Sat Fri - Sat [16, 60) 34, 16 , 59 Age < 0 or Age > 120 -1 , 121 Age [0, 16) 2, 0 , 15 Non-numeric value [email_address] Guest status Visitor Visitor Day Sun - Thu Mon, Sun, Thu Field Valid equivalent classes Valid representative values Invalid equivalent classes Invalid representative values
• 25.
  Example: Country ClubIII valid valid (boundary) invalid Invalid age 121 Visitor Fri 12 Invalid age -1 Student Fri 11 Invalid age [email_address] Member Mon 10 9 Thu Visitor 59 50 8 Sat Student 120 45 7 Sun Member 15 10 6 Thu Student 60 30 5 Sat Member 16 10 4 Sun Visitor 0 25 3 Mon Student 100 30 2 Fri Member 34 25 1 Mon Visitor 2 25 Test case ID Day Guest status Age Result