Proceedings of the 1991 Winter Simulation Conference

Barry L. Nelson, W. David Kelton, Gordon M. Clark (eds.)

SECRETS OF SUCCESSFUL SIMULATION STUDIES

Averill M. Law
Michael G. McComsa

Averill M. Law & Associates

P.0, BOX 40996
Tucson, Arizona 85717

Reprinted from Industrial Engineering magazine, May no assistance in these areas.

11990. Copyright 1990, Institute of Industrial Engineers,
25 Technology Park/Atlanta, Norcross, Georgia 30092. 2 INGREDIENTS REQUIRED FOR SUCCESS

ABSTRACT We have found that the following are important

elements of a successful simulation project:
In many “simulation studies” the primary focus is on
simulation software selection and “programming. ” We . Knowledge of simulation methodology,
believe, however, that only 30 to 40 percent of the total stochastic models of operations
effort in most successful simulation projects is actually research (e. g., queueing theory),
model coding. In this tutorial we discuss ten key steps probability theory, and statistics
that should, in fact, compose a sound simulation study.
● Formulating the problem correctly
:1 INTRODUCTION
● Obtaining “good” information on system operating
The use of simulation modeling to design new systems procedures and control logic
and to “fine-tune” the performance of existing systems
continues to increase at a rapid pace due to the increased ● Modeling system randomness in a reasonable
complexity of contemporary systems, reduced computing manner
costs, improvements in simulation software, and the
availability of animation. However, there still is the ● Choosing appropriate simulation software and
unfortunate impression that simulation is largely a utilizing it correctly
complicated exercise in computer programming. Thus,
in many simulation studies the major emphasis is . Establishing model validity and credibility
erroneously on simulation software selection (see Law
and Haider 1989) and on the “coding” of the simulation ● Using proper statistical procedures for interpreting
lprogram. simulation output (classical statistical techniques for
However, based on our simulation consulting independent data are not directly applicable)
~experiences during the past fourteen years, we believe
‘that simulation modeling is actually a sophisticated ● Employing good project management techniques
systems analysis activity, and that model coding
,represents only 30 to 40 percent of the total efort in a Simulation methodology, which includes such activities
typical sound simulation study. Even if a simulation as validation techniques, selecting simulation input
jpackage could be developed that made model coding probability distributions, and analyzing simulation output
time negligible, there would still be a number of data, is taught in most university industrial engineering
substantive project issues that would have to be and management science departments. It is also
addressed, as discussed in the next section. available in public short courses offered by several
Furthermore, current simulation software offers little or organizations.

21
22 Law and McComas

3 STEPS IN A SOUND SIMULATION STUDY ● Decide how the model will be used in the decision-
making process (e.g., on a one-time baais to make
In Figure 1 we show the steps that will compose a capital expenditures for a new system, or to make
typical, sound simulation study and the relationships weekly production-scheduling decisions)
between them. The number beside the symbol
representing each step refers to the more detailed ● Determine who will be the model’s end user (e.g.,
discussion of the step below. The amount of time that is an experienced programmer/analyst versus a
required for a particular step will depend on the system production engineer), since this affuta how user
being modeled; for example, there will generally be friendly the model must be
considerably more &ta to collect and analyze for an
existing system than for a proposed one. Some ● Specify measures of performance (e.g., mean daily
simulation projects may require steps that are not throughput) that will later be used by management
depicted in the diagram. Moreover, a simulation study to compare alternative system configurations, since
is not a simple sequential process. We may determine a model may be capable of providing an accurate
at some point in the modeling process (e.g., Steps 3 or estimate of one measure, but not another (see Pitfall
6) that our model is not completely “valid,” which will No. 9 in hW and Mc(!omas 1989)
require us to redefine our model or to collect more &ta
(Step 2). In certain cases, we might even have to ● Delineate the system configurations to be studied,
reformulate our project objectives (Step 1). to avoid major reprogramming later

Step 1. Foxmulate problem and plan the study

1
One of the most important, but often neglected, aspects
of a simulation study is a careful statement of the
project’s objectives. This is partly due to a lack of
understanding of the nature of simulation, the 2
information it can provide, and the time and effort
required for a sound study. (We recommend that a
knowledgeable simulation analyst conduct a one-hour 3 Valid No
?
seminar on these topics for relevant managers and
engineers, if appropriate.) + Yes

9
It is impossible to decide upon an appropriate level of Construct a
4 computer program
model detail without knowing precisely what issues are and verify
to be addressed by the model. We recommend that
project goals be set at an initial meeting that includes
5 Make pilot mns
managers, engineers, ad operational personnel.
However, one should not necessarily expect a single
simulation model to be capable of efficiently addressing Valid No
6
?
several widely disparate objectives. For example, due to
computer execution time considerations, it might be Yes

necessary to use one simulation model to study the

*
7
detailed workings of a particular subsystem, while
+
another more aggregate model would be used to explore
8
the effectiveness of the overall system.
The following tasks should be completed at the first
9
meeting:

Identify any performance problems for the cmisting 10

system (if there is one)

State definitively the study’s overall objectives and Figure 1: Steps in a simulation study
also five to ten very specific issues to be addressed
by the model
Secrets of Successful Simulation 23

After the above initial meeting has been completed, the analyst to interact with the manager (and other key
simulation analyst(s) should plan the overall study in project personnel) on a regular basis throughout the
terms of the number of people, the time, and the cost project. This approach has the following benefits:
required for each aspect of the project. It is our
eqxa-ience that simulation studies often take somewhat ● Often when a study is first initiated, there is not a
longer than expected, due to a poor initial understanding clear idea of the problem to be solved. ‘l%us, as
of the complexity of the system’s operating procedures. the study proceeds and the nature of the problem
becomes clearer, this information should be
Step 2. Collect data and define a model conveyed to the manager who may reformulate the
study’s objectives. The greatest model for the
The simulation analyst should collect information on wrong problem is clearly invalid!
system operating procedures and control logic. This will
generally not be an easy task, since no single person or . The manager’s interest and involvement in the
document will have the required information. Thus, for study are maintained.
example, in the case of a manufacturing system the
analyst might have to talk to such people as machine ● The manager’s knowledge of the system contributes
operators, industrial and manufacturing engineers, to the actual validity of the model.
production planners, managers, and vendors. This
process can be further complicated by inaccurate . The model is more credible, since the manager
information and by the lack of formalized system understands and accepts the model’s assumptions.
operating procedures. Data should be collected (if (How many managers would be willing to make a
possible) to specify model parameters (e.g., a conveyer million-dollar decision based on a model that they
speed) and input probability distributions (e. g., for do not understand or agree with the assumptions?)
machine operating and repair times). In general, each As a matter of fact, it is extremely desirable to
source of system randomness should be represented by have the manager and other important personnel
an appropriate probability distribution (notjust its mean) “sign off” on key model assumptions and to take
in the model (see Pitfall No. 7 in Law and McComas “ownership” of the model.
1989). Furthermore, the “correctness” of each
c[istribution should be evaluated by using graphical Step 3. Valid?
comparisons and statistical tests (see Law and Kelton
1.991, Chapter 6 and Law and Vincent 1991). A very important idea for validity/credibility
The above information and data (in summary form) enhancement is for the analyst to perform a structured
should be carefully delineated in what we call an walk-through of the conceptual model (as embodied in
“assumptions document” (see Law 199 1). This report the assumptions document) using an overhead projector
will typically be twenty pages in length for a before an audience of all key people. This helps to
manufacturing system. The assumptions document is ensure that the model’s assumptions are correct,
used in the structured walk-through of the conceptual complete, and consistent (i. e., that “local” information
model (Step 3) and is the main documentation for the obtained from different people is not contradictory).
model. At a typical structured walk-through, several erroneous
Data should also be collected on the performance of model assumptions are discovered and corrected, a few
the existing system (if possible) to aid in validating the new assumptions are added to the model, and some
model (see Step 6). level-of-detail issues are resolved by the system
The level of model detail should depend on project “experts” present. (The assumptions document should,
objectives, data availability, credibility concerns, of course, be up&ted to reflect these changes.)
computer constraints, and the opinions of system Furthermore, at the end of the meeting, it is common for
‘“experts. ” For example, models used to design new all people present to feel that ~ now have a valid
systems are generally less detailed than those used to model.
“optimize” existing systems, because of differences in The structured walk-through should be performed
lproject goals and in &ta availability. Note that it is before coding begins to avoid significant reprogramming
neither necessary nor desirable to have a one-to-one if major proble~ are discovered at the meeting.
correspondence between each element of the system and
leach element of the model.
Finally, it is extremely important for the simulation
24 Law and McComas

Step 4. Construct a computer program and verify Most simulation software vendors offer a version of
their software with animation capabilities. Animation is
The choice of the software used to develop the useful for communicating the essence of a simulation
simulation progmm can have a large impact on project model (or of simulation itself) to managers and other
success; it will effect the level of detail possible and, interested persons, which increases the credibility of the
thus, model validity (if the software is not flexible model. For models with a complicated flow of entities,
enough), model execution time, and project completion animation is also useful for debugging the program and
time. finding errors in the model’s logic. The following are,
There are two major classes of software used for however, two limitations of animation:
simulation modeling: general-pupose programming
languages and simuk.ztion packages (or sojbvare). . Animation is nota substitute for a carefil statistical
General-purpose languages (e.g., FORTRAN or C) are analysis of the simulation output data (Step 9)
usually already known by the analyst, are available for
all computers, and are less expensive to purchase. . A “correct” animation is no guarantee of a valid or
Simulation packages, on the other hand, reduce debugged model
programming time significantly, provide a natuml
framework for simulation modeling, and typically reduce There are a number of techniques used for debugging
overall project cost. In general, we believe that it is (also called ver~fving) the simulation program, including:
prudent for an organization to consider the use of
software designed specifically for simulation. . Developing the program in a modular manner
The two principal kinds of simulation packages are
simulation languages and applications-oriented . Using interactive debuggers and traces
simulators (see Law and Haider 1989 for details).
Simulation languages offer essentially unlimitedmodeling ● Performing a structured walk-through of the code
jkibility, but require programming expertise. Examples
of simulation languages are AutoMod II, GPSS H or ● Checking simulation output data for reasonableness
GPSS PC, MODSIM II, SIMAN IV, SIMSCRIPT 11.5,
and SLAM II. ● Animation
Simulators are currently available for certain types of
manufacturing, computer/communications, and Step 5. Make pilot runs
airport/airspsce systems. Their goal is to be able to
construct a simulation “program” by the use of menus Pilot runs of the “debugged” simulation model are
and graphics, without the need for programming. When made for validation purposes in Step 6.
simulators are applicable, they may require considerably
less program development time than a simulation Step 6. Valid?
language. They are also easier to learn and have
modeling constructs more closely related to the system Numerical results and animations for the pilot nny
of interest. lhe major drawback of many simulators is should be reviewed carefully by system experts to detect
that they are limited to modeling only those system remaining errors in the model assumptions, and the
configurations allowed by their standard features. This model modified to reflect any necessary changes. Note
difficulty can be partially overcome if the simulator also that a realistic animation can sometimes establish
contains “progmrnming-like commands” to model credibility for a simulation model or project almost
complex decision logic or if the simulator has the ability instantaneously. For example, a manager of operations
to call routines written in a general-purpose language. at one factory, who was unfamiliar with simulation
(Most of the model would still be developed using menus modeling, stated “That is my system!” upon seeing the
and graphics.) Examples of simulators are COMNET animation of his production line for the first time. His
11.5, LANNET 11.5, NETWORK 11.5, ProModel, interest in the simulation project was established from
SIMFACTORY 11.5, WITNESS, and XCELL+ . that time on.
We believe based on our simulation consulting Use sensitivity analyses (see Law and Kelton 1991, pp.
experiences that most valid simulation models of 310-3 11) to determine what model aspects (e.g., an input
“complex” systems will require programming of some parameter, a probability distribution, or the level of
sort, regardless of whether a simulation language or a detail for a subsystem) have the greatest impact on the
simulator is used. desired performance measures, Given that there is
Secre t~ of Successful Simulation 25

always only a limited amount of time and money for ● Number of independent simulation runs
model development, “sensitive” aspects of the model
should obvioual y be modeled the most carefidl y. For ● Initial conditions for each simulation run (e.g., all
example, we used sensitivity analysis to determine the machines idle and no parts present)
basic “unit of production” moving through a simulation
lmodel of a food packaging plant. We found that using ● Length of the warmup period, if one is appropriate
ii case of food items (approximately 500) rather than a
single food item as the model unit of production did not We recommend always making at least three to five
:iffect the simulation results appreciably, but reduced independent runs for each alternative, and using the
model execution time dramatically. average of the estimated performance measures from the
The most definitive test of the validity of a simulation individual runs as the overall estimate of the
model is establishing that its performance measures performance measure. (Independent rims means using
(closely approximate the performance measures that different random numbers for each run, starting each run
‘would be expected from a proposed system configuration in the same initial state, and resetting the model’s
((see Law and Kelton 1991, pp. 311-319). If a system statistical counters back to zero at the beginning of each
~similar to the proposed system now exists, then a pilot run.) This overall estimate should be more statistically
:simulation run is made for the existing system and its precise than the estimated performance measure from
output measures are compared to the corresponding one run.
measures for the existing system itself. If the two sets When simulating some systems (e.g., certain types of
of measures compare “closely, ” then the model of the manufacturing, computer, or communication systems),
misting system is considered “valid. ” The model is then we are often interested in the long-run (or steady-state)
modified so that it represents the proposed system. The behavior of the system, i.e., its behavior when operating
greater the commonality between the existing and in a “normal” manner. On the other hand, simulations
proposed systems, the greater our confidence in the of these kinds of systems often begin with the system in
model of the proposed system. There is, however, no an empty and idle (or some other unrepresentative) state.
completely definitive approach for validating the model This results in the output data from the beginning of the
of the proposed system. simulation not being representative of the desired
“normal” behavior of the system. Therefore, simulations
Step 7. Design experiments are often run for a certain amount of time, the warmup
period, before the output data are actually used to
It must be decided what system configurations to estimate the desired measure of performance. Use of
simulate, since there are sometimes more possible these warmup period data would bias the estimated
alternatives than one can reasonably simulate. Usually performance measure.
the complete decision cannot be made at this time, since
the analysis of the production runs in Step 9 typically Step 8. Make production runs
suggests additional system designs to simulate.
Since random samples from the input probability The simulation runs specified in Step 7 are executed
distributions “drive” a simulation model for a particular on a computer.
system configuration through time, basic simulation
output data (e.g., daily tbroughputs in a factory) or an Step 9. Analyxe output data
estimated performance measure computed from them
(e.g., average tiny throughput from the entire run) are The output data from the production runs are used to
also random. l?w.r, a simulation model only produces a constmct numerical estimates of the desired measures of
statistical estimate of the (true) pe~ormance measure, performance for each system configuration of interest.
not the measure itself (see Pitfall No. 10 in Law and (Confidence intervals can be used to determine the
McComaa 1989). In order for a simulation estimate to statistical precision of these estimates.) These estimated
be statistically precise (have a small variance) and free performance measures are then used to determine the
of bias (have mean equal to the performance measure), efficacy of particular system designs andlor to determine
the analyst must speci~ for each system design the best system design relative to the specified
appropriate choices for the following: performance measures (Step 1).
In addition to numerical e@imates of the performance
. Length of each simulation run measures, it is often useful to employ graphical displays
(histograms, pie and bar charts, time plots) of the
26 Law and McComas

simulation output data to gain further insights about decision-making process is the credibility of the
system behavior. For example, in Figure 2 we plot total simulation model (and the analyst). This is why we have
inventory (cases of product in process) as a function of emphasized such activities as regular interaction with
time for a simulated factory. This plot provides management, the structured walk-through of the
considerable information about the dynamic conceptual model, and the use of animation.
characteristics of the system, such as the warmup period
(the first three weeks) and the long-term cyclical nature 4 suMMARY
of the inventory level (caused by the system operating
procedures), Simulation modeling is a sophisticated systems analysis
Note that the simulation results for the system designs activity that requires technical knowledge and project
specified in Step 7 will often suggest additional management skills to develop a model that is both valid
alternatives to simulate. and actually used in the decision-making process.
Organizations that are embarking on their first
simulation project should probably obtain the assistance
of a simulation consultant, because of the high level of
TOTAL INVENTORY
t 14000 expertise required and the many potential pitfalls
awaiting the unwary simulation user. The consultant

‘!
should not only help with the details of the project, but
should also provide a technology transfer on simulation
methodology.

REFERENCES

Law, A.M. August 1991. Documenting Simulation

Models. Zhdustrial Engineering 23:15-16.
o
1 WEEK 20 I..aw, A.M. and S.W. Haider. 1989. Selecting
Simulation Software for Manufacturing Applications:
Practical Guidelines and Software Survey. Industrial
Figure 2: Plot of factory inventory level Engineering 21:33-46.
as a function of time I-.aw, A.M. and W.D. Kelton. 1991. Simulation
Modeling and Analysis, Second Edition. New York:
McGraw-Hill.
Step 10. Document, present, and implement results Law, A.M. and M.G. McComas. May 1989. Pitfalls to
Avoid in the Simulation of Manufacturing Systems.
Good documentation is very important, because Industrial Engineering 21:28-31.
simulation models are often used for more than one Law, A.M. and M.G. McComas. May 1990. Secrets of
application. It should include an assumptions document, Successful Simulation Studies. Industrial Engineering
documentation of the program, and a report summarizing 22; 47-48, 51-53, 72.
the results and conclusions of the study. The Law, A.M. and S.G. Vincent. 1991. UniFit 11 User’s
“concreteness” of the assumptions document will also Manual. Tucson, Arizomx Averill M. Law &
enhance the credibility of the model, and make the Associates.
analyst seem thorough and organized.
Most simulation projects end with a final presentation, AUTHOR BIOGRAPHIES
which is often attended by people (e.g., the plant
manager) who were not involved with the details of the AVERILL M. LAW is President of Averill M. Law&
model-building process. Thus, model credibility may Associates (Tucson, Arizona). He has been a simulation
have to be established for these people, and animation consultant to such organizations as General Motors,
will certainly be useful in this regard. It is also IBM, AT&T, ALCOA, General Electric, 3M, Nabisco,
important to discuss how information was obtained for Xerox, NASA, and the Army. He has presented more
the model and what efforts were made to validate and than 180 simulation seminars in 10 countries. He is the
verify the model. author (or coauthor) of three books and more than 30
One of the most important factors in determining papers on simulation, manufacturing, operations
whether the simulation results will actually be used in the research, and statistics, including the widely used
Secrets of Successful Simulation 27

textbook Simulation Modeling and Analysis. His series

of papers on the simulation of manufacturing systems
won the 1988 Institute of Industrial Engineers’ best
publication award. He is the codeveloper of the UniFit
II software package for fitting probability distributions to
observed data, and he developed a four-hour videotape
on simulation with the Society of Manufacturing
Engineers. Dr. Law writes a regular column on
simulation for lndu.wrial Engineering magazine. He has
taught simulation at the University of Arizona and the
University of Wisconsin. Dr. Law has a Ph.D. in
Industrial Engineering and Operations Research from the
University of California at Berkeley.

MICHAEL G. McCOMAS is Vice President for

Consulting Services of Averill M. Law & Associates.
He has considerable simulation modeling experience in
such manufacturing industries as food processing, paper
products, microcomputers, aerospace materials, medical
supplies, electronic components, pet care products, and
basic metal processing. His educational background
includes an M. S. in Systems and Industrial Engineering
from the University of Arizona. He is the coauthor of
three papers on manufacturing applications of
simulation.