Chapter Seven
Verification and validation Techniques
Topics to be covered
1. Model building, verification and validation 2. Verification of simulation model 3. Calibration and validation of models, - face validity, - validation of model assumption, - validating input-output transformations, - input output validation using historical data
1. Model building, verification and validation
Model
Rarely capture precisely system characteristics Acceptable model
Model correctness / goodness test
Tries to test if model is correct to desired level
Verification and validation
Are two most important tests of goodness Verification correctness of the formal representation of the intended model Validation test how realistic the model assumptions are Care should be taken- absence of evidence is not evidence
of absence
2. Verification of simulation model
Contains
Inspecting simulation program logic Performing simulation test runs and evaluating sample path trajectory Perform consistency checks
Verification through analysis of test runs
Test if simulation model correctly implements design specifications
1. input parameter and output statistics
Check if model entities are correctly flowing Count number of customers arriving and departing
2 using debugger
Specific to simulation program used
3. using Animation
Used for basic verification Using different entity pictures
Verification through test runs
Following entities through the simulation Changing entity pictures Displaying global variables or entity attributes Displaying plots of global variable or entity attributes Displaying levels of system statistics
4. Sanity checks
Check statistics of simulated values of arrival and departures with expected values
Verification via performance analysis
Performance analysis
Computation of performance measures and Verification of certain relations among them
Performance measures in queuing systems
Average number of jobs in queue and in system Average job waiting time Average system time Server utilization Throughput
Evaluate the measures analytically Estimate the steady state probability of number of jobs
Regenerative queuing system and busy cyles
each busy cycle
idle time Busy time
Complete cycle
Regeneration cycle- repeats it self
Let TB be busy period and TC be regeneration cycle If arrival is Expo( ), idle time is also Expo( ) If system capacity is infinite
number of jobs arriving = number of jobs departing
and
Regeneration cont
For exponential arrival
1 E(TB ) 1
Throughput
Is departure rate Average throughput is given by
Example of model verification
A single server system
Unif(1,8) arrival times (time in hours) Expo(4) service time ( time in hours) Capacity of 4 Simulated for 100,000hours Jobs completed 20,661 and 1578 rejected
Throughput
o1 20,661 0.20661 10,000
utilization 1 / 4.5 1/ 4 0.888
o2
1 (0.88) 0.22 4
Verification based on Littles formula
From simulation,
average number of customers in queue=1.37 Average number in system =2.2085
From Littles formula
average number of customers in queue
Ns
eff
Wb
(0.25) * (0.88) * (6.6325) 1.4738
Average number in system
Ns
eff
Wb
(0.25) * (0.88) * (6.6325 4) 2.233
3. Calibration and validation of models
Validation is critical to credibility of model Need for validation
Assumptions about input data and others Simplifications omitting processes Oversights process elements forgotten Limitations - human/software( general purpose/specific simulation language) / data limitations(input data)
Two types of validation
Face validity validity at the surface Statistical validity
Statistical validity
Collect data from existing system
Performance metrics Parameter values
Compare it with model outputs
If there is no significant difference, model is valid
3.1. face validity
Is validation at the surface Achieved with assistance of domain experts It is helpful
to avoid last minute rushes Identifies secondary objectives or interests
Should be a continuous process
Animations Special event handling
Necessary but not sufficient
Is subjective
3.3. validating input-output transformations Check if input data assumption is valid
Fitting data to theoretical distribution
Graphical approach Chi-square test Kolmogorove -Smirinov test Square error
Graphical approach
Create a histogram of the observed data Create a histogram of the theoretical dist Visually compare the graphs
Example: check if the following service time is from normal dist
Mean =5.61 St. dev=2.25 Count=30
10
Chi-square test
Preferred method of goodness test Steps
Establish null hypothesis Determine level of test significance Calculate the critical value from Chi-square dist Calculate the chi-square test statistics from the data Compare with critical value Accept or reject null hypothesis
Null hypothesis
This states that the data has come from the assumed distribution The alternative hypothesis is data did not come from the distribution
Example: if assumed distribution is normal Ho=norm( , ) H1=not norm( , )
Level of test significance
Is the level to which test error is tolerated If confidence level is 95%, test significance will be 0.05 Common levels 0.01, 0.1
Determine critical value for chi-square distribution
Boundary between significant halves distribution Example: significant and non of the chi-square
95% confidence, level of significance is 5% critical value-95% to the left and 5% to the right
Chi-square test statistics
Calculated as
where
Compare and accept or reject
Compare test statistics with the critical value Accept - If test statistics is less than level of significance
Data is from the assumed distribution
Reject if test statistics is greater than level of significance
Data is not from assumed distribution
Draw backs of Chi-square test
Needs sufficient quantity of data points At least 20 data points If data points are less, use KS test
Example: Chi-square test
Verify that the following inter arrival time of customers is exponentially distributed
Soln.
Data points are 30- chi-square test is possible Mean=2.31 Sd=2.88
Soln. cont
Hypothesis
Ho= Expo(2.31) H1=not Expo(2.31)
Level of significance
Let us use 95% confidence interval Level of significance 0.05
Critical value
Divide the data into cells
Equal width Equal probability- 6 cells
If there are J cells and N data points
Relative frequency of observations in cell j is
The theoretical probability distribution is
For continuous distribution
Determine critical value
Degree =number of cells-number of param-1 i.e. 6-1-1=4 Critical value = refer chi-square table at (0.05,4)
i.e. 9.49
Example cont
Calculate the test statistics table
The chi-square test statistics is
Compare with test statistics
accept (4<9.49) Distribution is exponential
3.4. Input output validation using historical data
Source of input data
Historical data Manufacturers spec Vendor claims Operator estimates Management estimates Automatic data capture Direct observation
If base system or similar one existed Avoids the need for real time data collection Risks
System to be simulated may not be identical Data is not collected for simulation
