Chapter 12

Comparison and Evaluation of

Alternative System Designs

Banks, Carson, Nelson & Nicol

Discrete-Event System Simulation
Purpose
 Purpose: comparison of alternative system designs.
 Approach: discuss a few of many statistical methods that
can be used to compare two or more system designs.
 Statistical analysis is needed to discover whether
observed differences are due to:
 Differences in design or,
 The random fluctuation inherent in the models.

2
Outline
 For two-system comparisons:
 Independent sampling.
 Correlated sampling (common random numbers).
 For multiple system comparisons:
 Bonferroni approach: confidence-interval estimation, screening,
and selecting the best.
 Metamodels

3
Comparison of Two System Designs
 Goal: compare two possible configurations of a system
 e.g., two possible ordering policies in a supply-chain system, two
possible scheduling rules in a job shop.
 Approach: the method of replications is used to analyze
the output data.
 The mean performance measure for system i is denoted
by i (i = 1,2).
 To obtain point and interval estimates for the difference in
mean performance, namely 1 – 2.

4
Comparison of Two System Designs
 Vehicle-safety inspection example:
 The station performs 3 jobs: (1) brake check, (2) headlight check,
and (3) steering check.
 Vehicles arrival: Possion with rate = 9.5/hour.
 Present system:
 Three stalls in parallel (one attendant makes all 3 inspections at
each stall).
 Service times for the 3 jobs: normally distributed with means 6.5, 6.0
and 5.5 minutes, respectively.
 Alternative system:
 Each attendant specializes in a single task, each vehicle will pass
through three work stations in series
 Mean service times for each job decreases by 10% (5.85, 5.4, and
4.95 minutes).
 Performance measure: mean response time per vehicle (total
time from vehicle arrival to its departure).
5
Comparison of Two System Designs
 From replication r of system i, the simulation analyst obtains an
estimate Yir of the mean performance measure i .
 Assuming that the estimators Yir are (at least approximately)
unbiased:
1 = E(Y1r ), r = 1, … , R1; 2 = E(Y2r ), r = 1, … , R2
 Goal: compute a confidence interval for 1 – 2 to compare the
two system designs
 Confidence interval for 1 – 2 (c.i.):
 If c.i. is totally to the left of 0, strong evidence for the hypothesis that
1 – 2 < 0 (1 < 2 ).
 If c.i. is totally to the right of 0, strong evidence for the hypothesis
that 1 – 2 > 0 (1 > 2 ).
 If c.i. is totally contains 0, no strong statistical evidence that one
system is better than the other
 If enough additional data were collected (i.e., Ri increased), the c.i.
would most likely shift, and definitely shrink in length, until conclusion of
1 < 2 or 1 > 2 would be drawn.
6
Comparison of Two System Designs
 In this chapter:
 A two-sided 100(1-)% c.i. for 1 – 2 always takes the form of:
Y.1  Y.2  t / 2, s.e.(Y.1  Y.2 )
where Y.i is the sample mean performance measure for system i over all replications,
and  is the degress of freedom,

 3 techniques discussed assume that the basic data Yir are

approximately normally distributed.

7
Comparison of Two System Designs
 Statistically significant versus practically significant
 Statistical significance: is the observed difference Y.1  Y.2 larger
than the variability in Y.1  Y.2 ?
 Practical significance: is the true difference 1 – 2 large enough
to matter for the decision we need to make?
 Confidence intervals do not answer the question of practical
significance directly, instead, they bound the true difference
within a range.

8
Independent Sampling with Equal Variances
[Comparison of 2 systems]

 Different and independent random number streams

are used to simulate the two systems
 All observations of simulated system 1 are statistically
independent of all the observations of simulated system 2.
 The variance of the sample mean, Y.i, is:
V  Y.i   i2
 
V Y.i 
Ri

Ri
, i  1,2

 For independent samples:

 12  22
V Y.1  Y.2   V Y.1   V Y.2   
R1 R2

9
Independent Sampling with Equal Variances
[Comparison of 2 systems]
 If it is reasonable to assume that  (approximately) or if
R1 = R2, a two-sample-t confidence-interval approach can be
used:
 The point estimate of the mean performance difference is:
Y.1  Y.2
 The sample variance
Ri
for system i is:
Ri

  2  R 1 1  Yri 2  RiY.i 2
1
S i2  Yri  Y.i
Ri  1 r 1 i r 1

 The pooled estimate of 2 is:

( R1  1) S12  ( R2  1) S 22
S 2p  , where   R1  R2 -2 degrees of freedom
R1  R2  2
 C.I. is given by: Y.1  Y.2  t / 2, s.e.(Y.1  Y.2 )

 Standard error:  
s.e. Y.1  Y.2  S p
1

1
R1 R2
10
Independent Sampling with Unequal
Variances [Comparison of 2 systems]

 If the assumption of equal variances cannot safely be

made, an approximate 100(1-)% c.i. for can be
computed as:
S12 S 22
 
s.e. Y.1  Y.2  
R1 R2

 With degrees of freedom:


S2
1 / R1  S 22 / R2 2
, round to an interger

 1 1
 1 
 S 2 / R 2 /  R  1    S 2 / R
  2 2
 2
/  R2  1 


 Minimum number of replications R1 > 7 and R2 > 7 is

recommended.

11
Common Random Numbers (CRN)
[Comparison of 2 systems]
 For each replication, the same random numbers are used to
simulate both systems.
 For each replication r, the two estimates, Yr1 and Yr2, are
correlated.
 However, independent streams of random numbers are used on
different replications, so the pairs (Yr1 ,Ys2 ) are mutually
independent.
 Purpose: induce positive correlation between Y.1,Y.2 (for each
r) to reduce variance in the point estimator of Y.1  Y.2 .
      
V Y.1  Y.2  V Y.1  V Y.2  2 cov Y.1 , Y.2 
 12  22 2 12 1 2 12is


   positive
R R R
 Variance of Y.1  Y.2 arising from CRN is less than that of
independent sampling (with R1=R2).
12
Common Random Numbers (CRN)
[Comparison of 2 systems]

 The estimator based on CRN is more precise, leading to

a shorter confidence interval for the difference.
 Sample variance of the differences D  Y.1  Y.2 :
1  
R R

 
1

2 2
S D2  Dr  D  2
Dr  RD
R  1 r 1 R  1  r 1 

R
1
where Dr  Yr1-Yr 2 and D 
R  D , with degress of freedom  R -1
r 1
r

 Standard error:

s.e.( D )  s.e. Y.1  Y.2   SD
R

13
Common Random Numbers (CRN)
[Comparison of 2 systems]

 It is never enough to simply use the same seed for the

random-number generator(s):
 The random numbers must be synchronized: each random
number used in one model for some purpose should be used for
the same purpose in the other model.
 e.g., if the ith random number is used to generate a service time
at work station 2 for the 5th arrival in model 1, the ith random
number should be used for the very same purpose in model 2.

14
Common Random Numbers (CRN)
[Comparison of 2 systems]
 Vehicle inspection example:
 4 input random variables:
 An, interarrival time between vehicles n and n+1,
 Sn(i), inspection time for task i for vehicle n in model 1 (i=1,2,3; refers
to brake, headlight and steering task, respectively).
 When using CRN:
 Same values should be generated for A1, A2, A3, …in both models.
 Mean service time for model 2 is 10% less. 2 possible approaches to
obtain the service times:
 Let Sn(i), be the service times generated for model 1, use: Sn(i) -
0.1E[Sn(i)]
 Let Zn(i), as the standard normal variate,  = 0.5 minutes, use:
E[Sn(i)] +  Zn(i)
 For synchronized runs: the service times for a vehicle were generated
at the instant of arrival and stored as its attribute and used as needed.

15
Common Random Numbers (CRN)
[Comparison of 2 systems]

 Vehicle inspection example (cont.): compare the two

systems using independent sampling and CRN where R = R1 =
R2 =10 (see Table 12.2 for results):
 Independent sampling: Y.1  Y.2  5.4 minutes

with   17, t 0.05,17  2.11, S12  118.0 and S 22  244.3, c.i. : -18.1  θ1-θ 2  7.3

 CRN without synchronization: Y.1  Y.2  1.9 minutes

with   9, t 0.05,9  2.26, S D2  208.9, c.i. : - 12.3  1 -  2  8.5

 CRN with synchronization: Y.1  Y.2  0.4 minutes

with   9, t 0.05,9  2.26, S D2  1.7, c.i. : - 0.50  1 -  2  1.30
 The upper bound indicates that system 2 is at almost 1.30 minutes
faster in expectation. Is such a difference practically significant?

16
CRN with Specified Precision
[Comparison of 2 systems]
 Goal: The error in our estimate of 1 – 2 to be less than  .
 Approach: determine the number of replications R such that the
half-width of c.i.: H  t / 2, s.e.Y.1  Y.2   
 Vehicle inspection example (cont.):
 R0 = 10, complete synchronization of random numbers yield 95%
c.i.: 0.4  0.9 minutes
 Suppose  = 0.5 minutes for practical significance, we know R is
2
the smallest integer satisfying R  R0 and:  t / 2, R 1S D 
R   
  
 Since t / 2, R 1  t / 2, R0 1 , a conservation estimate of R is:
2
 t / 2, R0 1S D 
R   

  
 Hence, 35 replications are needed (25 additional).
17
Comparison of Several System Designs
 To compare K alternative system designs
 Based on some specific performance measure, i, of system i , for i =
1, 2, …, K.
 Procedures are classified as:
 Fixed-sample-size procedures: predetermined sample size is used to
draw inferences via hypothesis tests of confidence intervals.
 Sequential sampling (multistage): more and more data are collected
until an estimator with a prespecified precision is achieved or until
one of several alternative hypotheses is selected.
 Some goals/approaches of system comparison:
 Estimation of each parameter ,.
 Comparison of each performance measure i, to control 1.
 All pairwise comparisons, i, - i, for all i not equal to j
 Selection of the best i.
18
Bonferroni Approach [Multiple Comparisons]

 To make statements about several parameters simultaneously,

(where all statements are true simultaneously).
 Bonferroni inequality:
C
P(all statements S i are true, i  1, ...,C)  1  
j 1
j  1 E

Overall error probability, provides an upper

bound on the probability of a false conclusion

 The smaller j is, the wider the jth confidence interval will be.
 Major advantage: inequality holds whether models are run with
independent sampling or CRN
 Major disadvantage: width of each individual interval increases
as the number of comparisons increases.

19
Bonferroni Approach [Multiple Comparisons]

 Should be used only when a small number of comparisons are

made
 Practical upper limit: about 20 comparisons
 3 possible applications:
 Individual c.i.’s: Construct a 100(1- j)% c.i. for parameter i,
where # of comparisons = K.
 Comparison to an existing system: Construct a 100(1- j)% c.i.
for parameter i- 1 (i = 2,3, …K), where # of comparisons = K – 1.
 All pairwise: For any 2 different system designs, construct a
100(1- j)% c.i. for parameter i- j. Hence, total # of comparisons
= K(K – 1)/2.

20
Bonferroni Approach to Selecting the Best
[Multiple Comparisons]
 Among K system designs, to find the best system
“Best” - the maximum expected performance, where the ith design
has expected performance i.
 Focus on parameters:  i  max j  i  j for i  1,2,..., K
 If system design i is the best, it is the difference in performance
between the best and the second best.
 If system design i is not the best, it is the difference between
system i and the best.
 Goal: the probability of selecting the best system is at least
1 – , whenever  i  max j i  j   .
 Hence, both the probability of correct selection 1-, and the
practically significant difference , are under our control.
 A two-stage procedure.
21
Bonferroni Approach to Selecting the Best
[Multiple Comparisons]
 Vehicle inspection example (cont.): Consider K = 4 different
designs for the inspection station.
 Goal: 95% confidence of selecting the best (with smallest
expected response time) where  = 2 minutes.
 A minimization problem: focus on  i  min j  i  j for i  1,2,..., K
  = 2, 1- = 0.95, R0=10 and t = t0.0167,9 = 2.508
 From Table 12.5, we know:
2 2 2 2 2 2
S12  4.498, S13  28.498, S14  5.498, S 23  11 .857, S 24  0.119, S 34  9.849
 The largest sample variance Sˆ 2  max i  j S ij2  S132  28.498 , hence,
  2.508 2 * 28.498  
R  max 10,  2    max10,  44.8   45
  2  
 Make 45 - 10 = 35 additional replications of each system.

22
Bonferroni Approach to Selecting the Best
[Multiple Comparisons]
 Vehicle inspection example (cont.):
 Calculate the overall sample means:
45
1
Yi 
45 Y
r 1
ri

 Select the system with smallest Y i is the best.

 Form the confidence intervals:

  
min 0, Y i  min j i Y j  2   i - min j i  j  max 0, Y i  min j i Y j  2 
 Note, for maximization problem:
The difference for comparison is:  i  max j i  j
   


 The c.i. is: min 0, Y i  max j i Y j  2   i - max j  i  j  max 0, Y i  max j i Y j  2

23
Bonferroni Approach for Screening
[Multiple Comparisons]
 A screening (subset selection) procedure is useful when a two-
stage procedure isn’t possible or when too many systems.
 Screening procedure: The retained subset contains the true
best system with probability  1- when the data are normally
distributed (independent sampling or CRN).
 Specify 1-, common sample size from each system and R2.
 Make R replications of system i to obtain Y1i, Y2i, … < YRi for system i =
1,2, …, K.
 Calculate the sample means for all systems Y .i.
2
 Calculate sample variance of the difference for every system pair S ij.
 If bigger is better, then retain system i in the selected subset if:
Sij
Y.i  Y. j  t /(K 1),R 1 for all j  i
R
 If smaller is better, then retain system i in the selected subset if:
Sij
Y.i  Y. j  t /(K 1),R 1 for all j  i
R 24
Metamodeling
 Goal: describe the relationship between variables and the
output response.
 The simulation output response variable, Y, is related to k
independent variables x1, x2, …, xk (the design variables).
 The true relationship between variables Y and x is represented
by a (complex) simulation model.
 Approximate the relationship by a simpler mathematical function
called a metamodel, some metamodel forms:
 Linear regression.
 Multiple linear regression.

25
Simple Linear Regression [Metamodeling]

 Suppose the true relationship between Y and x is

suspected to be linear, the expected value of Y for a
given x is: E(Y|x) = 0 + 1x
where 0 is the intercept on the Y axis, and 1 is the slope.

 Each observation of Y can be described by the model:

Y = 0 +  1 x + 
where is the random error with mean zero and constant variance 2

26
Simple Linear Regression [Metamodeling]

 Suppose there are n pairs observations, the method of

least squares is commonly used to estimate 0 and 1.
 The sum of squares of the deviation between the observations
and the regression line is minimized.

27
Simple Linear Regression [Metamodeling]

 The individual observation can be written as:

Y i = 0 + 1 x i +  i
where are assumed to be uncorrelated r.v.
 Rewrite:
Yi   0'  1(xi -x )   i

n
where  0'   0  1 x and x  x /n
i 1 i

 The least-square function (the sum of squares of the deviations):

L 
n
i 1
 i2   Y  
n
i 1 i 0  1 x i   
2 n
i 1
Yi   0'  1 ( xi  x )  2

'
 To minimize L, find L /  0 and L / 1 , set each to zero, and solve for:

n
n Yi S Yi ( xi  x )
ˆ0  Y 
'

and ˆ1 
xy
 i 1


i 1 n n
S xx ( xi  x ) 2
i 1
S xy – corrected sum of
S xx – corrected sum of squares of x
cross products of x and Y 28
Test for Significance of Regression
[Metamodeling]
 The adequacy of a simple linear relationship should be tested
prior to using the model.
 Testing whether the order of the model tentatively assumed is
correct, commonly called the “lack-of-fit” test.
 The adequacy of the assumptions that errors are NID(0,2) can
and should be checked by residual analysis.

29
Test for Significance of Regression
[Metamodeling]

 Hypothesis testing: H 0 : 1  0 and H 1 : 1  0

 Failure to reject H0 indicates no linear relationship between x and
Y.

 If H0 is rejected, implies that x can explain the variability in Y, but

there may be in higher-order terms.

Straight-line model is Higher-order term is

adequate necessary 30
Test for Significance of Regression
[Metamodeling]

 The appropriate test statistics:

ˆ1
t0 
MS E / S xx
 The mean squared error is:
ei2 S yy  ˆ1S xy

n
MS E  
i 1 n  2 n2
which is an unbiased estimator of 2 = V(i).
 t0 has the t-distribution with n-2 degrees of freedom.
 Reject H0 if |t0| > t/2, n-2.

31
Multiple Linear Regression [Metamodeling]

 Suppose simulation output Y has several independent variables

(decision variables). The possible relationship forms are:

Y = 0 + 1x1 + 2x2 + …+ mxm + 

Y = 0 + 1 x 1 + 2 x 2 + 

Y = 0 + 1x1 + 2x2 + 3x1x2 + 

32
Random-Number Assignment for
Regression [Metamodeling]

 Independent sampling:
 Assign a different seed or stream to different design points.
 Guarantees that the responses Y from different design points will
be significantly independent.
 CRN:
 Use the same random number seeds or streams for all of the
design points.
 A fairer comparison among design points (subjected to the same
experimental conditions)
 Typically reduces variance of estimators of slope parameters, but
increases variance of intercept parameter

33
Optimization via Simulation
 Optimization usually deals with problems with certainty, but in
stochastic discrete-event simulation, the result of any simulation run
is a random variable.
 Let x1,x2,…,xm be the m controllable design variables & Y(x1,x2,…,xm)
be the observed simulation output performance on one run:
 To optimize Y(x1,x2,…,xm) with respect to x1,x2,…,xm is to maximize or
minimize the mathematical expectation (long-run average) of
performance, E[Y(x1,x2,…,xm)].

 Example: select the material handling system that has the best
chance of costing less than $D to purchase and operate.
 Objective: maximize Pr(Y(x1,x2,…,xm)D).
 Define a new performance measure:
1, if Y(x1,x 2 ,...x m )  D
Y ' ( x1,x 2 ,...x m )  
0, otherwise
 Maximize E(Y’(x1,x2,…,xm)) instead.
34
Robust Heuristics
[Optimization via Simulation]
 The most common algorithms found in commercial optimization
via simulation software.
 Effective on difficult, practical problems.
 However, do not guarantee finding the optimal solution.
 Example: genetic algorithms and tabu search.
 It is important to control the sampling variability.

35
Control sampling variability
[Optimization via Simulation]
 To determine how much sampling (replications or run length) to
undertaken at each potential solution.
 Ideally, sampling should increase as heuristic closes in on the
better solutions.
 If specific and fixed number of replications per solution is required,
analyst should:
 Conduct preliminary experiment.
 Simulate several designs (some at extremes of the solution space
and some nearer the center).
 Compare the apparent best and apparent worst of these designs.
 Find the minimum for the number of replications required to declare
these designs to be statistically significantly different.
 After completion of optimization run, perform a 2nd set of experiments
on the top 5 to 10 designs identified by the heuristic, rigorously
evaluate which are the best or near-best of these designs.
36
Summary
 Basic introduction to comparative evaluation of
alternative system design:
 Emphasized comparisons based on confidence intervals.
 Discussed the differences and implementation of independent
sampling and common random numbers.
 Introduced concept of metamodels.

37