Test Sequence Generation with Cayley Graphs (Talk @ A-MOST 2021)

S. Hallé, R. Khoury
Sylvain Hallé and Raphaël Khoury
Université du Québec à Chicoutimi
CANADA
Test Sequence Generation with
Cayley Graphs
CRSNG
NSERC
17th Workshop on Advances in Model Based Testing (A-MOST), April 12th, 2021

Model-based testing for reactive systems

system

system
reactive
receives inputs ("actions")
accepts or rejects inputs
stateful

system
reactive
stateful
speciﬁcation

system
reactive
stateful
state-based
incomplete
formal speciﬁcation

, ,
, ✓
test

, ,
, ✓
test sequence
ﬁnite
associated outcome

, ,
, ✓
test sequence
ﬁnite
associated outcome
test
generation

, ,
, ✓
test sequence
ﬁnite
associated outcome
test
generation
model-based
sequence

Formal model
Let...
Q be a set of observable states
A be a set of actions

Formal model
Let...
1
3
4
2
A reactive speciﬁcation is a Moore machine

Formal model
Let...
1
3
4
2
a
b
b
b
c
d
with
input alphabet A

Formal model
Let...
1
3
4
2
a
b
b
b
c
d
with
input alphabet A
?
⊤
⊤
⊤
and output alphabet {⊤, ⊥, ?}

Goal: ﬁnd sequences that "exercise" the
speciﬁcation in various ways
Intuition

Intuition
Never press right after

Intuition
If you press or , then next

= coverage
Intuition
If you press or , then next

FSM-based coverage criteria
1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
What does "covering the spec" mean? Is it...
visiting each state?
executing each action?
taking each transition?
ending in each state?
...or in each transition?
visiting each triplet of states?
a, db
ab, dc
ab, dc, dbb
a, ab, d, dc
a, ab, d, dc, db, dbb
aba, abcd, dbbd,
dbba
. . . . . . . . .
. . . . . .
. . . . . .
. . . . . . .
. . . . .

FSM-based coverage criteria
1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
What does "covering the spec" mean? Is it...
visiting each state?
executing each action?
taking each transition?
ending in each state?
...or in each transition?
visiting each triplet of states?
a, db
ab, dc
ab, dc, dbb
a, ab, d, dc
a, ab, d, dc, db, dbb
aba, abcd, dbbd,
dbba
. . . . . . . . .
. . . . . .
. . . . . .
. . . . . . .
. . . . .
One distinct algorithm
for each criterion

Contribution
We propose a theoretical framework where:
each coverage criterion becomes a particular
case of a general deﬁnition
a test suite achieving full coverage can be
generated by the same algorithm for all criteria
Key concepts:
triaging function
Cayley graph

Triaging function
Given a speciﬁcation M, a test is a sequence:
q0 q1 qn
a0
...
a1 an−1
i.e. a ﬁnite word over Σ ⊆ (Q × A × Q)*.

Triaging function
Given a speciﬁcation M, a test is a sequence:
q0 q1 qn
a0
...
a1 an−1
i.e. a ﬁnite word over Σ ⊆ (Q × A × Q)*. A triaging
function is any surjective function
κ : Σ → C
"categories"

A triaging function κ induces an equivalence
relation ~ over test sequences
Σ
Triaging function

Triaging function
σ ~ σ' iﬀ κ(σ) = κ(σ')
Σ

A test suite covers
a triaging function
if it contains one
sequence in each
category.
σ ~ σ' iﬀ κ(σ) = κ(σ')
Σ
Triaging function

Coverage ratio
A numerical value can be associated to a set of test
sequences ΣT ⊆ Σ:
ρ(ΣT) =
{κ(σ)}
∪
σ∈ΣT
C

Coverage ratio
A numerical value can be associated to a set of test
sequences ΣT ⊆ Σ:
ρ(ΣT) =
{κ(σ)}
∪
σ∈ΣT
C
categories
covered
all categories

Examples
Different coverage criteria correspond to different
definitions of κ.
κ( ) = ...
q0 q1 qn
a0
...
a1 an−1

Examples
deﬁnitions of κ.
κ( ) = ...
q0 q1 qn
a0
...
a1 an−1
qn

Examples
deﬁnitions of κ.
κ( ) = ...
q0 q1 qn
a0
...
a1 an−1
qn state residual coverage

Examples
deﬁnitions of κ.
κ( ) = ...
q0 q1 qn
a0
...
a1 an−1
qn
an−1
state residual coverage
action residual coverage

Examples
deﬁnitions of κ.
κ( ) = ...
q0 q1 qn
a0
...
a1 an−1
qn
an−1
qn−1 qn
transition residual coverage
an−1

Examples
deﬁnitions of κ.
κ( ) = ...
q0 q1 qn
a0
...
a1 an−1
qn
an−1
qn−1 qn
n
length coverage
an−1

Prefix closure
We may also want to consider as "covered"
categories generated by all prefixes of a sequence.
The prefix closure of κ is the function ∫κ defined as:
∫κ(σ) = {κ(σ')}
∪
σ' σ
Output values of a prefix closure are called families
(i.e. sets of categories).

Examples (prefix closure)
Taking the preﬁx closure of the functions deﬁned
earlier produce other coverage criteria.
κ( ) = ...
q0 q1 qn
a0
...
a1 an−1
qn
an−1
qn−1 qn
an−1

Examples (prefix closure)
Taking the preﬁx closure of the functions deﬁned
earlier produce other coverage criteria.
κ( ) = ...
q0 q1 qn
a0
...
a1 an−1
{q0, ..., qn}
{a0, ..., an−1}
{q0 q1, ...,
qn−1 qn
an−1
∫
}
a0
...and a few others (see the paper)

Test sequence generation
Problem
Solution
Given a triaging function κ over an alphabet
A, generate a set of sequences that
completely covers κ
1. Generate the Cayley graph of κ
2. Pick important vertices of the graph
3. Compute the corresponding Steiner tree

Step 1: Cayley graph
Given a triaging function κ : Σ → C, a Cayley graph
is a Moore machine Gκ such that...
Σ is the input alphabet
C is the output alphabet
for every sequence σ,
the output of Gκ is κ(σ)
In other words, Gκ "encodes" the triaging function.

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
# d
d
b
a
b
c
b
a
b
a
d
d
c
a
1
3
a
b
4
b
b
2
c
d
state
residual
action
residual
b

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
# d
d
b
a
b
c
b
a
b
a
d
d
c
a
1
3
a
b
4
b
b
2
c
d
state
residual
action
residual
b
1
#
1
d c a b

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
# d
d
b
a
b
c
b
a
b
a
d
d
c
a
1
3
a
b
4
b
b
2
c
d
state
residual
action
residual
b
2
d
d
d
2
d
d c a b

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
# d
d
b
a
b
c
b
a
b
a
d
d
c
a
1
3
a
b
4
b
b
2
c
d
state
residual
action
residual
b
1
c
c
c
1
c
d c a b

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
# d
d
b
a
b
c
b
a
b
a
d
d
c
a
1
3
a
b
4
b
b
2
c
d
state
residual
action
residual
b
3
a
a
a
3
a
d c a b

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
# d
d
b
a
b
c
b
a
b
a
d
d
c
a
1
3
a
b
4
b
b
2
c
d
state
residual
action
residual
b
1
b
b
b
1
b
d c a b

For a given κ and alphabet A, Gκ can be generated
by an exploratory algorithm. States are triplets of
the form (c,q,a).

the form (c,q,a).
For each state (c,q,a)...

the form (c,q,a).
category
sequence
state of M

the form (c,q,a).
For each a ∈ A...
category
sequence
state of M

the form (c,q,a).
For each a ∈ A...
q' = δ(q,a)
c' = κ(a a)
// Next state in M
// Category of a a
category
sequence
state of M

the form (c,q,a).
For each a ∈ A...
q' = δ(q,a)
c' = κ(a a)
If a state (c',q',a') already exists
Link (c,q,a) (c',q',a')
a
// Next state in M
// Category of a a
category
sequence
state of M

the form (c,q,a).
For each a ∈ A...
q' = δ(q,a)
c' = κ(a a)
If a state (c',q',a') already exists
Link (c,q,a) (c',q',a')
Else
Create state (c',q',a a)
Link (c,q,a) (c',q',a a)
a
a
// Next state in M
// Category of a a
category
sequence
state of M

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
κ(ε) = #

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
κ(ε) = #
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
κ(a) = a
δ(1,a) = 3
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
a
a 3 a
κ(a) = a
δ(1,a) = 3
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
a
a 3 a
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
a
a 3 a
κ(d) = d
δ(1,d) = 2
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
d
d 2 d
a
a 3 a
κ(d) = d
δ(1,d) = 2
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
d
d 2 d
a
a 3 a
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
d
d 2 d
a
a 3 a
κ(ab) = b
δ(3,b) = 1
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
d
d 2 d
b
b 1 ab
a
a 3 a
κ(ab) = b
δ(3,b) = 1
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
d
d 2 d
b
b 1 ab
a
a 3 a
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
d
d 2 d
b
b 1 ab
a
a 3 a
κ(dc) = c
δ(2,c) = 1
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
c
c 1 dc
d
d 2 d
b
b 1 ab
a
a 3 a
κ(dc) = c
δ(2,c) = 1
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
c
c 1 dc
d
d 2 d
b
b 1 ab
a
a 3 a
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
c
c 1 dc
d
d 2 d
b
b 1 ab
a
a 3 a
κ(dcd) = d
δ(1,d) = 2
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
d
c
c 1 dc
d
d 2 d
b
b 1 ab
a
a 3 a
κ(dcd) = d
δ(1,d) = 2
# 1 ε

1
3
4
2
a
b
b
b
c
d
?
⊤
⊤
⊤
Example
a
b
d
b
a
b 4 db
d
c
c 1 dc
d
d 2 d
b
b 1 ab
a
a 3 a
# 1 ε

Step 2: picking vertices of Gκ
# d
d
b
a
b
c
b
a
b
a
d
d
c
a
1
3
a
b
4
b
b
2
c
d
state
residual
action
residual
b
The resulting graph can contain multiple states
with the same label. Pick a set VI of important
vertices that contains each label once.

Step 2: picking vertices of Gκ
# d
d
b
a
b
c
b
a
b
a
d
d
c
a
1
3
a
b
4
b
b
2
c
d
state
residual
action
residual
b
The resulting graph can contain multiple states
with the same label. Pick a set VI of important
vertices that contains each label once.
How to select "best" set is left open...

Step 3: find a Steiner tree
# d
d
b
a
b
c
b
a
b
a
d
d
c
a
1
3
a
b
4
b
b
2
c
d
state
residual
action
residual
b
A (minimal) Steiner tree is a subgraph of Gκ that
includes all the VI.

# d
d
a
b
c
c
a
1
3
a
4
b
2
d
state
residual
action
residual
b

# d
d
a
b
c
c
a
1
3
a
4
b
2
d
state
residual
action
residual
b
Use the tree to generate paths that end in each
element of VI.
d, db, a d, dc, db, a

Dealing with transitive closure
Given a triaging function κ, the previous
construction guarantees full coverage for both κ
and ∫κ.
⇒ optimal if the labeling of Gκ is injective

Dealing with transitive closure
Given a triaging function κ, the previous
construction guarantees full coverage for both κ
and ∫κ.
⇒ optimal if the labeling of Gκ is injective
We can do better for ∫κ!
Post-processing step: elimiate sequences
that provide no new coverage
state
residual
action
residual
d, db, a d, dc, db, a

Implementation and experiments
Technique concretely implemented as an extension
of SealTest
Compared to GraphWalker and original SealTest
Sample of 33 reactive speciﬁcations
44 variants of coverage criteria

Implementation and experiments
Technique concretely implemented as an extension
of SealTest
Compared to GraphWalker and original SealTest
Sample of 33 reactive speciﬁcations
44 variants of coverage criteria
GW and ST fail to achieve full coverage
in 23% and 38% of problem instances
Cayley generates tests suites that are about
4× smaller than both tools
But test suites contain about 2× more
sequences
Results:

Key result
Generating a (minimal) test suite for
a coverage criterion
Finding a (minimal) coverage on
its Cayley graph
=

Take-home points
Multiple coverage criteria handled in a uniform
formal framework
What other criteria can be modeled in this way?
The algorithm for generating Gκ is (probably)
optimal*
The procedure to generate a test suite from Gκ is
deﬁnitely not
Results apply to all coverage criteria at once

Take-home points
Multiple coverage criteria handled in a uniform
formal framework
What other criteria can be modeled in this way?
The algorithm for generating Gκ is (probably)
optimal*
The procedure to generate a test suite from Gκ is
deﬁnitely not
Results apply to all coverage criteria at once
Thank you!

Test Sequence Generation with Cayley Graphs (Talk @ A-MOST 2021)

More Related Content

Similar to Test Sequence Generation with Cayley Graphs (Talk @ A-MOST 2021)

More from Sylvain Hallé

Recently uploaded

Test Sequence Generation with Cayley Graphs (Talk @ A-MOST 2021)