5 暂时将你眼睛闭了起来~Testing the Software with Blinders on

Dynamic, Black-box Testing

• Obviously, we need something that defines the software—
i.e.
requirements document or specification document.
We must be able to specify what should be produced as output B, given input A.
• We do NOT need to know HOW output B is produced from input A.
•Although we can’t look at code, we may ask the programmers questions about numeric limits, internal quirks of the software etc. (More on this shortly.)

测试方法：Test-to-Pass & Test-to-Fail

导致成功的测试&导致失败的测试

• Test-to-Pass: Assure only that the software minimally works, don’t push its capabilities, don’t see what you can do to break it
• Test-to-Fail or Error-forcing: designing and running test cases with the sole purpose of breaking the software, is strategically chosen to probe for common weaknesses in the software.
• 为什么要提出这两种测试？
  Use Test-to-Pass to reveal bugs before you Test-to-Fail
  首先Test-to-Pass确保被测程序可测

等价类Define equivalence partitions - or classes

研究对象：input,output
目的：Equivalence partitioning helps us cut down the number of test cases without adding a great deal of risk.

• Normally we look at two areas for testing:
  • Data
  • Logic flow
    等价类是指测试相同目标或者暴露相同软件缺陷的一组测试案例。
    在寻找等价类时，想办法把软件的相似输入、输出、操作分成组。这些组就是等价类。
    There are many guidelines for picking equivalence partitions.
    Note: These are guidelines, not hard and fast rules.

数据测试Data Testing

----Guidlines FOR CHOOSING EQUIVALENCE CLASSES
Identify boundary conditions 识别边界条件——data that is at the edge of the planned operational limits of the software.
For the boundaries, partition input into

• Valid data inside the boundary.
• Data just on the boundary (which may be valid or invalid).
• Invalid data just outside the boundary limits.
  Boundary conditions should be identified in the requirements or the specifications.

Equivalence Partitioning 1

Idea is to partition the input space into a number of equivalence classes such that one could expect, based on the specification, that every element of a given class would be ‘‘handled’’ (i.e., mapped to an Output) in the same manner (either correctly or incorrectly), thus reducing the total number of test cases that must be developed.
Two types of classes are identified:
valid (corresponding to inputs deemed valid from the specification) and
invaild** (corresponding to inputs deemed erroneous from the specification

Equivalence Partitioning 2

• Divide all possible inputs into classes (partitions) such that : There is a finite number of input equivalence classes
• You may reasonably assume that
  The program behaves: same as inputs in the same class
  One test with a representative value from a class is sufficient
  If the representative detects a fault, then other class members would detect the same fault.

Equivalence Classes

Valid data
• User supplied commands.
• Responses to system prompts.
• File names.
• Computational data:
• Physical parameters,
• Bounding values, and
• Initiation values.
• Output data formatting commands.
• Responses to error messages.
• Graphical data.
Invalid data
• Data outside bounds of the program.
• Physically impossible data.
• Proper value supplied in wrong place.

Equivalence Classes Strategy

• Identify input equivalence classes
  Based on conditions on inputs/outputs in specification/description: Both valid and invalid input equivalence classes
  Based on heuristics探索 and experience :
  “input x in [1…10]” --> classes : x< 1, 1 $\le$ x $\ge$ 10, x > 10
  “enumeration 枚举A, B, C“ --> classes : A, B, C, not{A,B,C,}
  "input intege --> classes : n not an integer,
  n < min, min $\le$ n $\ge$ 0, 0 $\le$ n $\ge$ max, n> max ……
• Define one/couple of test cases for each class
  Test cases that cover valid classes
  Test cases that cover at most one invalid class

Equivalence Partitioning Example 1

Partition system inputs into groups (partitions) that should cause equivalent behaviour, include both valid and invalid inputs.
If input is a 5-digit integer between 10,000 and 99,999,
Equivalence partitions are:
< 10,000
10,000 - 99,999
> 99,999

Equivalence Partitioning Example 2

Identify (multiple sets of disjoint) equivalence classes for the following program specification fragment.
City Tax Specification:

1. If gross pay is no more than $30,000, the tax is 1%.
2. If gross pay is more than $30,000, but no more than $50,000, the tax is 5%.
3. If gross pay is more than $50,000, the tax is 15%.

Equivalence Partitioning 3

1. If input is a range, one valid and two invalid equivalence classes:
   
2. If input is a specific value, one valid and two invalid equivalence classes:
   
3. If input is a set of related values, one valid and one invalid class:
   
4. If input is Boolean, one valid and one invalid class:
   

Equivalence Partitioning Example 3

Test for addition operation of MS calculator.
Class 1: Integer
Class 2: Decimal fraction
Class 3: Negative
Class 4: Invalid input

Further Partitioning Possibilities

Check default, empty, blank, null, zero, and no data.
If nothing is entered, what happens?
Default set?
Error message?
Hung program?
Check invalid, wrong, incorrect, and garbage data.

• Users WILL use the software incorrectly.
• Data loses or crashes are blamed on the software,
  always — not the user
• Have fun with this one — devious, tricky, and nasty are good traits for a tester!

Black-Box testing - Boundary Value Analysis (BVA)

A technique based on identifying, and generating test cases to explore boundary conditions.
Boundary conditions are an extremely rich source of errors.
Natural language based specifications of boundaries are often ambiguous, as in ‘‘for input values of X between 0 and 40,…’’
May be applied to both input and output conditions.
Also applicable to white box testing.

• more errors occur at the boundaries of an input domain than in the "center”

Boundary Values 边界值

Based on experience / heuristics :

• Testing boundary conditions of eq. classes is more effective
  ie. values directly on, above, and beneath edges of classes
• Choose input boundary values as tests in input classes instead of, or additional to arbitrary values
• Choose also inputs that invoke output boundary values
  ( values on the boundary of output classes )
  Example strategy as extension of equivalence partitioning:
  choose one (n) arbitrary value(s) in each eq. class
  choose values exactly on lower /upper boundaries of eq. class
  choose values immediately below /above each boundary ( if applicable )

Guidelines for Identifying Boundary Values

1. If the input range is bounded by a and b, then use a, b, and values just above and just below a and b, respectively.
   
2. If the input is a number of values, use the minimum and maximum of the values and values just above and just below them, respectively.
   
3. Apply guidelines 1 and 2 to create output values at the minimum and maximum expected values.
4. If data structures have boundaries, test these boundary values and values just above and just below them, respectively.
   e.g., for an array of bounds 1 to 10 — test array index equal to 0, 1,2, 9, 10, 11

BVA Example 1

Test a function for calculation of absolute value of an integer
Valid equivalence classes :

• Test cases :
  class x < 0, arbitrary value: x = -10
  class x >= 0, arbitrary value x = 100
  classes x < 0, x >= 0, on boundary : x = 0
  classes x < 0, x >= 0, below and above: x = -1, x = 1

BVA Example 2

Test a function which limit user input to 6-digit positive integer

• Test cases :
  Class arbitrary value: X1 = 123123
  Class boundary value: X2 = 12345
  Class boundary value: X3 = 1234567
  Class boundary value: X4 = 1
  Class boundary value: X5 = 0
  Class invalid value: X6 = -123123
  Class invalid value: X7 = asdasd
  Others?

Class invalid value: X8 = 000123
X9 = asd123
X10 = Empty

Other types in Boundary

Numeric/Character/Position/Quantity/Speed/Location/Size
---->First/last, Min/Max, Star/Finish, Empty/Full, Slower/Faster, Largest/Smallest, Over/Under, Shortest/Longest … …

Examples of possible boundary conditions

P75
Limits on input data – size, type
Limits on output data- size, type
Look for limits such as
• First-1/Last+1
• Start-1/Finish+1
• Min-1/max+1
• Less than empty/ more than full
• just Over/Just Under
•etc.
Don’t hesitate to choose data outside boundaries, even when it seems absurd.

Guidelines for choosing Equivalence classes----For Data Testing

• Identify sub-boundary conditions or internal boundary conditions
• Some limits may not be apparent to the general end user.
• To locate some of these, you may need to talk with programmers.
Example: Powers-of-two boundary conditions

An Example: A flight simulator

Assume the user can control the height of the plane, determine what those limits are:
• Not specified?
• Why not?
• Nothing in a computer is infinite!
Try flying
• Below ground
• In outer space
• Through a mountain (or some other object)

ASCII Table

ASCII Encoding Always Produce Sub-Boundary Conditions

The ASCII code is 8 bits.
• If numeric digits are to be input, check just out of range with the ‘/’ and ‘:’
• If capital letters are to be input, check just out of range with the ‘@’ and ‘[‘
•If lower case letters are to be input, check just out of range with the forward quote and the ‘{‘.
NOTE: The error in using the default settings in the PowerPoint software. In each case, I encased the character in a single quote— i.e. the key below “.

Text-box Field

Default
Empty
Blank
Null
Zero
None

Garbage Data in Test-to-Fail

Invalid
Wrong
Incorrect
Garbage data

5.5 STATE TESTING状态测试

Testing the Software’s Logic Flow

All software flows from one state to another.
解决——等价类划分
apply equivalence partition techniques to the selection of the states and paths

For some software, these state changes are obvious:
Paint Program
To determine states, we need to build a state transition diagram or state transition map.
• Good requirements or specifications often include these.
• There are software tools for drawing them.
• Different diagramming techniques exist — we’ll use just one, but it is not the only one possible.

1. Create a State Transition Map状态迁移图

Such a map may be provided as part of the product specification. If it is, you should statically test it as described in Chapter 4, “Examining the Specification.” If you don’t have a state map, you’ll need to create one.

A STM is a graph showing the logic flow from state to state in the program. This is diagrammed using the symbols:

All of these are labeled
• States- to identify them
• Transitions- to identify what triggers movement from one state to another state. # Example of Part of STM

Obviously, these can become quite large!!

What the STM Shows

• Each unique state of the software.
软件可以处于的每一个唯一状态。 一个好的经验法则是，如果你不确定某物是否是一个单独的状态，它可能是。 如果你后来发现它不是，可以把它折叠成另一种状态，
• Input or condition needed to move from one state to the next.
从一个状态到下一个状态的输入或条件。
这可能是按键，菜单选择，传感器输入，电话铃声等等。 一个状态不能无缘无故地退出。 具体的原因是你在这里寻找的。
• What is set or what output is produced when a state is entered or exited.
这将包括一个菜单和按钮被显示，一个标志被设置，一个打印输出，一个计算正在执行，等等
Obviously, we can’t investigate all possible paths through the state transition map (—this is really the “traveling salesperson problem”.

note

Because you are performing black-box testing, you don’t need to know what low level variables are being set in the code. Create your map from the user’s view of the software.

2. Reducing the Number of States and Transitions to Test

Use Equivalence Partitioning to Choose Test Cases for State Testing

Possible choices for partitioning:

• Try to visit each state at least once.
• Test the most common or popular state-to-state transitions.
• Test the least common paths between states.
• Test all entrances and exits from error states.
• Test random state transitions — i.e. throw darts a the transition state map!
All of these are testing-to-pass cases.

What to Specifically Test 具体测试什么

•Involves checking all state variables which define a state.

State variables can be invisible but very important.

•The text suggests checking with the spec writers and programmers to identify possible states — but, caution DO NOT GO TO THE CODE LEVEL!
•Programmers will tend to want to drag out code if it exists!

Testing-to-Fail State Testing

Race Conditions(资源竞争) and Bad Timing

Check race conditions – a timing problem causes the execution to not proceed as planned.

• Try interrupting the program in the middle of its execution as see what happens. - Is data lost?
• Can the program be restarted in a clean condition?
• Start two instances of the same program and input to both.

Repetition, Stress & Load Testing （ 重复，压力和负荷）

Repetition testing can often find memory leaks泄漏 — memory not freed 释放completely when it should be.

Stress testing tries to run under bad conditions （boundary condition testing）
— low memory, slow CPU, etc.

Load testing overloads the program with huge data files, long periods of execution, etc.

Round all of this out by just playing like a dumb user! (But, call them an inexperienced user, please!)