14 Civil Aviation

Flight Simulators and Training

Ronald J. Lofaro and Kevin M. Smith

Contents

Introduction..................................................................................................................................... 258
An Overview of Civil Aviation Training, Flight Simulators, .
and the Human Factors Therein................................................................................................. 259
Introduction......................................................................................................................... 259
Flight Simulator Basics........................................................................................................ 261
Human Factors and Flight Simulators................................................................................. 262
Major Drivers in Civil Aviation Pilot/Crew Training......................................................... 263
Flight Simulators and Flight Training Devices..............................................................................264
Overview..............................................................................................................................264
Flight Simulators and Training............................................................................................ 265
Flight Training Devices and Training................................................................................. 265
Flight Simulator and FTD Assessment...............................................................................266
SFAR 58 and AQP.......................................................................................................................... 267
History................................................................................................................................. 267
SFAR/AQP: Overview and Synopsis................................................................................... 267
SFAR 58............................................................................................................................... 268
AQP, LOS/LOFT, and Simulators....................................................................................... 268
Line-Oriented Flight Training (LOFT).......................................................................................... 269
Background.......................................................................................................................... 269
Maximizing LOFT: The Mission Performance Model and the Operational .
Decision Making Paradigm........................................................................................................ 270
LOFT: Current and Future................................................................................................... 270
Risk Identification and Management: Training and Evaluation with MPM .
and ODM Paradigm............................................................................................................. 270
Mission Performance Model............................................................................................... 271
From CRM/MPM to ODM................................................................................................. 273
Copyright © 2008. Taylor & Francis Group. All rights reserved.

The ODM........................................................................................................................................ 274

Operational Decision Making: New Window of Opportunity............................................ 274
Introduction to the ODM Paradigm..................................................................................... 274
Components of the ODM Model......................................................................................... 275
Risk Management: Risk, Rising Risk, and Flying the Line................................................ 275
Risk Management: The Operational Envelope, Risk Location, and Cumulative Effect..... 275
Risk Management: Critical Mission Impact Areas and Their Components....................... 276
Boundary Conditions as Mission Critical Impact Areas..................................................... 277

 The views and opinions expressed in this chapter are solely those of the authors. They do not necessarily represent those
of any governmental, public, or private organizations.

257

72838.indb
Human Factors257in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:47 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 258 Human Factors in Simulation and Training

Risk Management: Situation Knowledge and Risk Location.............................................. 277

Risk Management: Rising Risk, Risk Location, and Response Action.............................. 277
A Distinction: Cumulative Effect Resultant versus Additive Effect Resultant................... 279
Operational Envelope and Rising Risk Continuum: Relationship...........................280
Putting It All Together: Risk Location and Risk Management in .
the Operational Envelope................................................................................................280
ODM and MPM in LOFT Design, Development, and Evaluation................................................. 281
Introduction......................................................................................................................... 281
LOFT: ODM and MRM...................................................................................................... 281
LOFT Design: Another Approach....................................................................................... 283
Training...............................................................................................................................284
References....................................................................................................................................... 285
Federal Aviation Administration Advisory Circulars (AC) and Regulations................................ 286

IntroductioN
The use of flight simulators for civil pilot/crew training and performance evaluation has evolved to
the point where they have become key and indispensable tools for air carrier education—and will
continue to be so. This is due to a conjunction of factors: safety, cost, simulator fidelity, and fairly
recent changes and additions to the federal aviation regulations (FARs). A short bit of explanation
on the Federal Aviation Administration (FAA) and FARs follows. The FAA regulates aviation in the
United States, from air traffic to civil aviation security, the operation of air carriers, pilot training,
and more. The federal regulations covering aviation are all found in Combined Federal Regulations
(CFR), Title 14, Aeronautics and Space; these are commonly referred to as the FARs. The FARs are
divided into parts (1 through 199) and each part has a descriptive title and is a specific and detailed
regulation. For example, CFR 14, Part 121 is the FAR titled Operating Requirement: Domestic
Flight and Supplemental Operations. The FARs are usually referred to by part number; this one
would just be called Part 121.
As an aid to aviation, the FAA normally published advisory circulars (ACs) that relate to
specific FARs. The ACs are designed to provide assistance and guidelines in complying with
FARs. In fact, these ACs, which often are longer than the length of the FAR, are written in
meticulous detail and serve as a “how to” template. The ACs are titled, numbered, and grouped
by such areas as air traffic control, civil and general aviation security, pilot training, and so on.
For example, AC 120-45A is the AC titled Airplane Flight Training Device Qualification. The
120 specifies the general area of air carrier and commercial operations and helicopters, whereas
the 45 refers to the fact that this is the 45th AC under the 120 area. The “A” means it is the first
revision.
For the remainder of the chapter we will refer to the FARs by part number only, and the same for
Copyright © 2008. Taylor & Francis Group. All rights reserved.

ACs. A caveat: We will, somewhat extensively, cite from FARs and ACs. As these are large docu-
ments, and we have space limitations, we will sometimes omit sentences within a relevant portion
of a citation. We may, occasionally, rather than use a long citation, synopsize what we are referring
to. (For the interested reader, the complete FARs and ACs can be found on the FAA website: http://
www.faa.gov.)
We will begin this chapter with an overview of flight simulators (FS) and their use in the air
carrier arena. From there, this chapter will proceed to a brief look at the human factors issues
and problems in civil aviation’s use of flight simulators. These are, in the main, the same issues
and problems of flight simulator use in any environment: fidelity, part-task trainers, transfer of
training, motion axes, transition training and the like. In addressing the human factors of civil
aviation, we recognize that the hexapod axial motion bases and the veridicality of the FS to the
actual flying environment are what have led to the use of the FS as the major pilot/crew train-
ing, evaluation and certification tool. As one example, flight simulators today have assumed such

72838.indb
Human Factors258in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:47 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 259

an important place in air carrier training that an aircraft-type rating can be obtained (almost)
entirely in a flight simulator. Advances in the capabilities of this generation of flight simulator has
led to modified FAA classification schemas (levels) for airplane flight simulators and flight train-
ing devices (FTDs) and to the first new pilot/crew training effort in over 30 years, the Advanced
Qualification Program (AQP).
Our human factors focus will be on training, and the models for training and evaluating the
skills we see as paramount for the pilot/crew in accomplishing risk identification and risk manage-
ment. Thus, the chapter will deal extensively with the use and maximization of line-oriented flight
training (LOFT). The Air Transport Association of America (ATA) and the FAA have worked
on the best ways to design, develop, and implement LOFT and line-oriented evaluation (LOE) in
flight simulators, as shown by the relevant FARs and ACs. The result is that LOFT, which is done
entirely in a flight simulator, has become, over the past 15 to 17 years, what could be called the
“crown jewel” of air carrier pilot/crew training. LOFT and LOE received initial impetus from their
use in crew resource management (CRM) training. However, CRM in the mid-1990s encountered
problems, with one result being the “Big Three” U.S. flag air carriers (American, Delta, and United)
almost completely revamping and renaming their CRM programs (Aviation Week and Space Tech-
nology, 1996).
LOFT, always recognized as the key element in CRM, has thus become more and more recog-
nized as the independent and indispensable training and evaluation tool for pilot/crew performance.
The FAA’s emphasis on air carrier crews going to a new training and certification paradigm, the
above-mentioned AQP as spelled out in (Special FAR) SFAR 58, has further enhanced the role of
LOFT and LOE.
Lastly, this chapter will deal with LOFT design, development, and use in crew performance and
evaluation, using the mission performance model (MPM) and the operational decision ­making (ODM)
paradigm. We will lay out the use of LOFT for learning/practice in operational decision making,
which results in risk assessment and reduction. This, the authors have long ­considered as the key
functions for any airline captain and crew.

An Overview of Civil Aviation Training,

Flight Simulators, and the Human Factors Therein
Introduction
There are three somewhat obvious benefits from the use of a flight simulator in training. These are
the underpinning of the now-extensive use of flight simulators in general aviation, civil aviation, and
military aviation. Briefly put, they are:

1. Cost reduction and increased efficiency by replacing the real system, the plane, with the
Copyright © 2008. Taylor & Francis Group. All rights reserved.

flight simulator.
2. Reduction in the hazards of training in the plane. The loss of life and injuries that result
from training accidents and incidents are well documented.
3. The ability to train skills and performances that cannot be trained in the aircraft, such as
malfunctions and adverse conditions and, more important, missions/tasks that may never
be performed in real-world operations but are essential components of the operational mis-
sion profile of the aircraft (cf. Flexman and Stark, 1987).

The extensive use of flight simulators in the civil aviation world has resulted from both recognizing
these benefits and a confluence of other causes. The first, as said, is that the flight simulator is a safe
environment, putting neither crew nor planes at risk. However, although crew and aircraft safety
is foremost, looking further we see the following set of converging vectors: In the formative years,
and continuing well into the late 1980s and early 1990s, both civil aviation and the FAA relied on

72838.indb
Human Factors259in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:48 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 260 Human Factors in Simulation and Training

ex-military members (mainly active-duty personnel who left after fulfilling their commitment, but also
some retirees in their 40s) for a source of experienced manpower. In civil aviation, these former
military members included pilots and aviation maintenance technicians (AMT). The same was
true in the FAA, where there was, perhaps, a higher emphasis placed on the ex-military pilot, who
could be placed in the flight standards, aircraft, and pilot certification areas without missing a beat.
The rationale for the aviation industry seeking (and welcoming) military personnel is both obvious
and subtle. It is obvious that the military was a source of highly trained and qualified personnel.
Plus, their training was both extensive and standardized. It is also true that there is a “brotherhood
of airmen,” where inclusion is highly dependent on airmen background, experiences, training and
even common acquaintances. Add to these factors a common “language,” one that is technical,
acronym-laden and replete with idiomatic expressions. All of the above are still active (albeit to a
lesser degree) today. One result of the influx of military personnel was the use of the flight simulator
as the major training tool for aircrew.
In passing, it must be remembered that the earliest flight simulator, with replicated cockpit
instrumentation, controls, and most aspects of flight built by Edward A. Link in 1929, was a
generic flight simulator for General Aviation (GA) use. This Link trainer, whose picture we
have all seen (it looks like a child’s drawing of a plane mounted on a base), soon evolved into an
instrument flight trainer as a result of WWII. From there, the great advances in flight simulators
had to do with its use in military training. The history of the early flight simulator development
of both capability and capacity for training was for use in training military pilots/aircrew. It
was natural, as many of the early air carrier and FAA personnel had military backgrounds,
that the use of the flight simulator in training and certification became paramount. Add to this
a piece of reality that is often overlooked: the air carriers simply do not have enough planes to
take significant or even small item numbers out of service for pilot training. As one example,
the largest U.S. flag carriers own/lease less than 300 planes each. Unlike the military, which
has large numbers of aircraft that are dedicated only to training pilots, the air carriers must use
their planes, in the main, to generate income. Considering that the air carrier’s run 24/7 sched-
ules, it is apparent that the flight simulator is, must be, and will continue to be the training and
­certification tool in civil aviation.
To resume: This military use of the flight simulator drove much of early flight simulator tech-
nology as aircraft became more complex, more automated, and achieved higher performance. On
the civil aviation front, the air carriers began to insist that the delivery of a flight simulator for a
new type of aircraft be simultaneous, or even before, the new aircraft was put in everyday service.
Similarly, the civil side of aviation demanded more and more simulator capability (fidelity and
veridicality). The major air carriers used their pilot and crew training facilities to house a grow-
ing number of flight simulators. Companies that made flight simulators worked closely with both
the airframe manufacturers and the air carriers to design and deliver flight simulators that met
Copyright © 2008. Taylor & Francis Group. All rights reserved.

the changing needs of pilot training and certification. Today, for example, we have civil aviation
flight simulators for the B-757/767 and 777, which cost upward of $35 million and cost in the
thousands of dollars per hour when in operation. These operational costs include maintenance,
the simulator operator, and the air conditioning needed to maintain the temperature in the flight
simulator faculty at a level that does not impact the highly sophisticated computers that drive the
flight simulator.
The final vector, or piece, of the equation has been the technological advances in flight simula-
tor capability during the 1980s and ’90s. The advances in the fidelity of the visual scene presented
to the crew, as well the fidelity of the response of flight simulator instruments (be they “glass” or
“steam”) to inputs by the crew are outstanding examples of this fidelity.
In summation, through a convergence of causal vectors, the period from approximately 1970
to today has seen the emergence of the flight simulator as the essential tool for pilot training and
certification in the civil area.

72838.indb
Human Factors260in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:48 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 261

Flight Simulator Basics

The primary functions of any flight simulator (i.e., the functional definitions, as it were) are:

1. To present information that the real system would present for the purpose of training
2. To provide a practice environment that facilitates and enhances the skills and knowledge of the
pilot and thus provides learning that enhances performance in the real system, the airplane

Put into other terms, a flight simulator is a system designed to “imitate” the functions of another sys-
tem (a plane) in a real operational environment and to be a realistic substitute that responds realisti-
cally to flightcrew inputs. The key here is that a flight simulator can be programmed to offer varied
experiences to a flightcrew, but experiences that are safe, in that if you “crash” in a simulator, there
is no injury, save to your pride. (We will return to the role of experience in flight training and flying
later in this chapter.) Basically, a flight simulator is a training device that is safer, less expensive,
capable of quick modification, and can operate in all weather and for all or any part of a 24-hour
day. The flight simulator presents accurate cockpit displays to the flightcrew and accurately (except
for complete motion capability) responds to control and avionics inputs—all the time processing
and storing data on the crew’s control inputs, etc.
The characteristics of a flight simulator are that it:

1. Stores data that can be replayed and analyzed on crew input/crew response
2. Stores data that can be used to generate a realistic “environment”/mission or portion;
control and other responses to crew inputs
3. Displays such data/info both to the flightcrew and to the flight simulator operator
4. Responds to crew inputs accurately as to their effects on both system and environment
5. As would the actual plane, has accurate/valid displays of the status of onboard systems/
components (e.g., EPRs) that are so vital for the crew to see and monitor
6. Provides two-way training interfaces for the flight simulator instructor and the flightcrew
being trained

There are research applications of different types of flight simulators. The engineering development
flight simulator is used in the cockpit/flight control systems design phase. This flight simulator
makes system design easier because it is quickly reconfigurable, so that one can conduct system
experiments on changes and reconfigurations without having to build or tear down a real system.
The research/engineering simulator is a system used to help with basic R&D and applied
research on system operations (to look at differences in various flight-control systems; e.g., fly
by wire vs. manual vs. fly by light). This flight simulator has many of the capabilities mentioned
for the engineering development flight simulator. It can also evaluate human performance limits
Copyright © 2008. Taylor & Francis Group. All rights reserved.

in the system and evaluate system interaction with other systems (data-link, dispatch, ATC). And
lastly, the research/engineering flight simulator can be used to train personnel in the operation of
the system.
Although, as mentioned, there are several types of flight simulators, the focus in this chapter will
be what is often termed the “full-up” flight simulator. Simply put, a “full-up” flight simulator has:

1. Three-axis motion base (pitch/roll/yaw) in two directions each (hexapod motion, six
degrees of freedom)
2. Computer-generated graphic displays for the “out the windows” visual scene; these include
most types of weather and environments, as well as night scenarios
3. A complete cockpit that is the same as in the aircraft type that the flight simulator models
4. Tremendous computer memory, which allows for superb capability in realistic flight
simulation

72838.indb
Human Factors261in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:49 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 262 Human Factors in Simulation and Training

In short, the flight simulator has ultra-realism, except damping as to the motion bases. These flight
simulator capabilities are the component parts of the team fidelity—the closeness to which the flight
simulator mimics actual flight. What may be of more import is the flight simulator’s veridicality
(see following subsection).

Human Factors and Flight Simulators

The human factors training functions that a flight simulator can address are many and varied. We
will simply list many of them. Such a list must include briefings and demonstrations, practice,
performance analysis, learning enhancement, providing knowledge of results of actions, providing
supplementary cues to the flightcrew, building cognitive structures, performance assessment, and,
finally, providing a safe environment for introducing adverse conditions, malfunctions, and outright
mechanical and other failures (cf. Flexman and Stark, 1987).
Human factors training as a major component of pilot training became possible with the advent
of AQP. This was the first time that a complete review of traditional training became possible. No
longer were air carriers tied to the existing FARs, but were allowed to build programs to meet more
specifically targeted needs. This training usually included CRM, but more accurately, allowed cur-
ricula development that addressed complex problem-solving events that a flightcrew would encoun-
ter operating under adverse conditions. Thus, one saw an explosion of “line-oriented” training and
certification events as part of this new emphasis on human factors.
There are many issues, human factors being a major one, in the building and use of flight simu-
lators for the training and evaluation of pilot/crew performance. We have briefly mentioned some,
although we did not identify them as issues, in the preceding subsection. These issues cluster around
the level of flight simulator fidelity necessary for training and/or evaluating a pilot/crew task. This
is not to say that other concerns do not exist. But, in the main, the use in training and evaluations
of a flight simulator deals what is called fidelity (of the flight simulator to the aircraft) in terms of
(a) the motion bases, (b) the out–the-windscreen/window visuals, and (c) the simulator response to
control inputs. All these issues can be subsumed under the more general question of “How veridi-
cal to actual flight does a flight simulator have to be in order to insure training that fully prepares a
pilot/crew for actually flying the plane?”
Veridicality is the closeness of the correspondence of the knowledge structures formed by
using the flight simulator (learning and using controls/input responses/instrument responses/
visual scene/motion, etc.) to the information environment it represents, that is, the actual aircraft
type. Because the flight simulator is used in training to build knowledge structures in the crew
that will be used in actual flight, it is obvious that veridicality is the primary factor in flight
simulator design and use. The higher the fidelity of a flight simulator, the more veridical the
knowledge structures built in the flight simulator are—making the flightcrew optimally prepared
for actual flight.
Copyright © 2008. Taylor & Francis Group. All rights reserved.

To be sure, we do not want to give the impression that other issues do not exist and could include
incorrect control inputs, incorrect sequencing, poor or incorrect decision making, and more. How-
ever, these are beyond the scope of this chapter.
We will, however, look at one basic human factors/training assumption: The use of any flight
simulator for training tasks and skills results in the skills gained transferring to the actual cock-
pit, called “transfer of training.” The assumption is that the knowledge structures and information
acquired previously on one task affects (neutrally) the ability to be trained on another/other tasks.
Although we are talking of tasks here, the intent is that the learning of tasks trained in the flight
simulator will aid the learning (performance) of a new task in the plane. Confusing as this may
sound, the task learned in the flight simulator, when performed in the plane, is referred to as a new
task. Why? Because the task in the plane, even though it is the same as the task in the flight simula-
tor, is being done now in the plane; therefore, it is called a new task. Note: Transfer of training also
occurs from plane to flight simulator.

72838.indb
Human Factors262in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:50 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 263

There are two types of training transfer that occur. The desired one is called positive trans-
fer where previous training/experience aids “learning” of new tasks. However, there also exists
negative transfer where previous experience interferes with learning new tasks or performing the
trained, “old” task in new environment. It is important again to note that positive and negative
transfer can occur in either direction: flight simulator to plane or plane to flight simulator. There are
some examples of skills gained in flying that are not transferrable to the cockpit. This would seem to
have to do with the veridicality of the flight simulator to the aircraft and can present some difficult
problems. It would also seem to have to do with how the flight simulator was certified for use.
As one component of transfer of training, there has been great emphasis on the need for and
value of the flight simulator having the capability for hexapod axial motion. At this writing (2006),
the FAA, NASA, and the DOT/Volpe Transportation Center were conducting sophisticated experi-
ments on the benefits and need for hexapod motion for the platform on which the airline flight
simulator is mounted. The ultimate goals of these experiments (Burki-Cohen, Go et al., 2003) are to
provide information for a possible FAA AC and to develop information for a possible FAA policy on
flight simulator motion requirements in airline pilot training and evaluation. A brief overview of the
research findings to date is that hexapod platform motion has a significant positive effect for flight-
crew evaluation, but no significant benefit for training. Further, certain enhancements to the motion
washout filters (lateral side force and heave motion) seem to be beneficial in all cases. However, for
more complete information, the reader is referred to the works cited above, as it is not our purpose
here to engage in a lengthy discussion of flight simulator motion issues and research.
Another problem is that of “simulator sickness”: a phenomenon in which a pilot becomes sick
(vertigo, nausea) in the flight simulator. This has been handled in several ways. It now seems clear that
it is caused by a complex interaction that involves conflicting visual and kinesthetic cues. Another
part of the interaction seems to be the time duration between flying and using the flight simulator.
It would appear that, for some individuals, use of the flight simulator in close proximity to hav-
ing actually flown can cause simulator sickness. This is (usually) easily remedied by specifying a
minimum time duration that must pass between flight and flight simulator. In some cases, there
has been a reverse “simulation sickness” reaction whereby the pilot becomes sick in the aircraft.
Obviously, this is dangerous, as well as possibly career ending. To the author’s knowledge, this has
been handled via specifying (as before) minimum time duration between flight simulator and flight.
Although medication, ranging from over-the-counter motion sickness pills to prescription anti-.
vertigo drugs can be used, it would seem to be only a one-time or short-term remedy. The rationale
here seems obvious.
We now will leave the above issues as we are quite sure that they are well covered in other chap-
ters of this text and, the issues/problems that we mention in passing are not the focus of the chapter.

Major Drivers in Civil Aviation Pilot/Crew Training

Copyright © 2008. Taylor & Francis Group. All rights reserved.

Aircrew training for air carrier pilots (Part 121) has evolved over the years due to five main drivers.
They are:

1. Technical advancements in aircraft systems and simulation realism

2. Engine out operations
3. Mission critical alerts and warnings
4. Adverse condition operations
5. Human factors

Technical advancements in aircraft systems have, for obvious reasons, driven pilot training pro-
grams. The features of a new system and how it should be used operationally have always been built
into the curriculum. A good example of this is terrain collision avoidance system, version II (TCAS
II), a hard/software system that provides the aircrew with alerts for terrain collision avoidance.

72838.indb
Human Factors263in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:50 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 264 Human Factors in Simulation and Training

Although TCAS I was initially introduced with a part-task trainer, it is now an important feature of
full-mission simulators, and collision avoidance training is now possible.
Engine out operations as a major driver of pilot training is not quite so obvious. With the advent
of multiengine aircraft, engine out training had always been important to pilot certification. How-
ever, with the advent of swept-wing turbojet aircraft, this training took center stage. This was due to
the unique aerodynamic properties of swept-wing aircraft, more specifically, asymmetrical thrust
and axis coupling. During asymmetrical thrust operations, the swept-wing turbojet aircraft experi-
ences pronounced axis coupling, manifested in a rapid roll off or wing drop along with an equally
pronounced yaw. Increased pilot skill was and is the only counter tactic to this potentially fatal
condition.
In 1968 a DC-8 training accident involving asymmetrical thrust led to the effort to conduct all
training in simulators. Such an effort was successful, prompting, among other things, advancements
in simulator realism.
Although often overlooked, the third major pilot training and certification driver has and con-
tinues to be mission critical alerts and warnings. This area includes such maneuvers as stalls and
steep turns, wind shear recovery, and ground proximity warning recovery. Recent additions include
CAT III auto land system failure recovery maneuvers. It is important to note that all of these recov-
ery maneuvers require the aircraft to be “hand-flown” by the pilot.
This last statement brings to the surface what we think is the major challenge facing training
managers today: the tension line between increasingly sophisticated autopilot systems and the con-
tinuing and pressing need for a high degree of basic “stick and rudder” pilot skills. Pilots therefore
need to demonstrate proficiency in (1) adverse conditions (which include engine out operations), (2)
low visibility operations, (3) mission critical alerts and warnings, and (4) system and human limita-
tions. We shall later show how these four conditions form the boundaries for the pilot’s worldview
and how they can be incorporated into a training/operation model to manage/reduce risk.

Flight Simulators and Flight Training Devices

Before we begin this and following sections, we again feel pressed to make the following caveat.
Much of the following is from FAA documents: SFAR 58 and its accompanying AC (AC 120-AQP);
AC 120-40B; AC 120-45A; AC 120-35B; and AC 1210-45B. For brevity, “ease of flow,” and resultant
clarity, we have condensed some of the material in these; we omitted portions not pertinent to this
chapter and often deleted references to other FARs and ACs; on some occasions, there is paraphras-
ing. As has been said, the complete documents are online if the reader wants to see the entirety of
any of the FAR and AC cited.

Overview
Copyright © 2008. Taylor & Francis Group. All rights reserved.

The availability of advanced technology has permitted greater use of flight simulators for training
and checking of flightcrew members. The complexity, costs, and operating environment of modern
aircraft also have encouraged broader use of advanced simulation. Simulators can provide more in-
depth training than can be accomplished in airplanes and provide a very high transfer of learning
and behavior from the simulator to the airplanes. The use of simulators in lieu of airplanes results
in safer flight training and cost reductions for the operators. It also achieves fuel conservation and
reduction in adverse environmental effects.
As technology progressed and the capabilities of flight simulation were recognized, FAR revi-
sions were made to permit the increased use of simulators in approved training programs. Simula-
tors have been used in training and some checking programs since the middle 1950s. Various FAR
amendments gradually permitted additional simulator use in training and checking aircrews. A
significant recognition of simulator capability has occurred since the early 1970s. In December
1973, FAR Amendments 61-62 and 121-108 permitted additional use of visual simulators. In the

72838.indb
Human Factors264in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:51 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 265

early 1990s, various ACs and SFAR 58 further recognized simulator capability and use in training
and evaluating flightcrews.
Of importance is the fact that the FAA makes a distinction between an airplane (flight) simula-
tor and an airplane flight training device. The FAA AC that deals with airplane simulators is FAA
AC 120-40B, and AC 120-45B deals with FTDs.
The term FTD covers everything from a PC with training specific software to a mock-up of
an instrument panel, to a complete cockpit. However, what air carrier pilots/crew use for our focal
point, LOFT, is a full-up, hexapod axial motion-based, high quality visual scene flight simulator,
often called the “box” or the “sim.” Although FTDs are used as part-task trainers, it is the sim alone
that is used for LOFT. The box has full mission capabilities to include ATC chatter/instructions as
well as day/night and various weather and wind conditions. The simulator operator can program
a mission (a flight from point A to point B) that introduces the full spectrum of conditions and
problems gleaned from the experiences and reports of other pilots who have flown that particular
route. The mission simulation also introduces conditions and problems that have been encountered
or reported on other flights/routes.

Flight Simulators and Training

An Airplane Simulator [commonly called a flight simulator] is a full size replica of an airplane’s instru-
ments, equipment, panels, and controls in an open flight deck area or an enclosed airplane cockpit,
including the assemblage of equipment and computer software programs necessary to represent the
airplane in ground and flight operations, a visual system providing an out-of-the cockpit view, a force
[motion] cueing system with provides cues at least equivalent to that of a three degree of freedom
motion system; and is in compliance with the minimum standards for a Level A simulator specified in
AC 120-40, as amended.
The airplane simulators are placed, graded as it were, into 4 levels, A through D; the FTDs are
similarly ordered, except the classification scheme of levels 1 through 7 is used. In both cases, the
levels refer to the capabilities and complexities (hard and soft ware) of the training equipment. In both
cases, all equipment are placed in a matrix, by level, that indicates what flight tasks can be trained at
each level.
The new designations and their relationships with the simulator definitions used and in FAR Part
121, Appendix H are:

Level A—Visual
Level B—Phase I
Level C—Phase II
Level D—Phase III

While trying not to oversimplify this distinction, the main difference is that a “full-up” airplane
Copyright © 2008. Taylor & Francis Group. All rights reserved.

simulator has axial motion capability whereas an FTD does not. This will become clearer next as
we give the FAA definitions of both types of flight training equipment.

Flight Training Devices and Training

An airplane flight training device is a full scale replica of an airplane’s instruments, equipment,
panels, and controls in an open flight deck area or an enclosed airplane cockpit, including the
assemblage of equipment and computer software programs necessary to represent the airplane in
ground and flight conditions to the extent of the systems installed in the device. An FTD does not
require a force (motion) cueing or visual system and meets the criteria outlined in the AC for a
specific flight training device level. In an FTD, any flight training event or flight checking event can
be accomplished
Nonvisual simulators are now grouped with Level 6 training devices, but must meet the require-
ments, except for visual, of a Level A simulator. There is no other change in their characteristics

72838.indb
Human Factors265in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:51 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 266 Human Factors in Simulation and Training

or description; just their name. Alphabetic designations were chosen for simulators to maintain a
distinction from the numerically designated training devices.
In coordination with a broad cross section of the aviation industry, the FAA has defined
seven levels of flight training devices as Level 1 through Level 7. Level 1 is currently reserved.
Levels 2 and 3 are generic in that they are representative of no specific airplane cockpit and
do not require reference to a specific airplane. Levels 4 through 7 represent a specific cock-
pit for the airplane represented. Within the generic or specific category, every higher level of
flight training device is progressively more complex. Because of the increase in complexity and
more demanding standards when progressing from Level 2 to Level 7, there is a continuum of
technical definition across those levels. (Note: For complete matrices of flight simulator and
FTD/levels and the tasks that can be trained and/or checked in each device, see AC120-40B and
AC120-AQP.)

Flight Simulator and FTD Assessment

The need for standard flight simulator and FTD assessment and qualification criteria was neces-
sitated by the use of simulators for training and checking. The evolution of the simulator technol-
ogy and the concomitant increase in permitted use has required a similar evolution of the criteria
for simulator qualification. A listing of known simulator criteria should be, therefore, informative.
The qualification basis for a given simulator may be any of the past criteria, depending on when the
simulator was first approved or last upgraded.
The training and checking credits for nonvisual and visual simulators were delineated in
FAR Part 61, Appendix A, and FAR Part 121, Appendices E and F. Four levels of simulators were
addressed; Basic (nonvisual and visual simulators), Phase I, Phase II, and Phase III. (These desig-
nations have since been replaced by levels A through D as seen in Subsection A.) Each of the four
levels is progressively more complex than the preceding level and each contains all the features of
preceding levels plus the requirements for the designated level. As the technology has advanced,
so has the qualification guidance. Efforts to keep the criteria updated are, therefore, ongoing with
active participation from both industry and government resources
Any FTD or airplane flight simulator must be assessed in those areas that are essential to
accomplishing airman training and checking events. The assessment requirements and guidelines
are, essentially, the same for both FTD and flight simulator. This includes climb, cruise, descent,
approach, and landing phases of flight. Crew member station checks, instructor station functions,
checks, and certain additional requirements depending on the complexity of the device (i.e., touch-
activated cathode ray tube instructor controls; automatic lesson plan operation; selected mode of
operation for “fly-by-wire” airplanes, etc.) must be thoroughly assessed. Should a motion system or
visual system be contemplated for installation on any level of flight training device, the operator
or the manufacturer should contact the network services project management (NSPM) for infor-
Copyright © 2008. Taylor & Francis Group. All rights reserved.

mation regarding an acceptable method for measuring motion and/or visual system operation and
applicable tolerances. The motion and visual systems, if installed, will be evaluated to ensure their
proper operation.
The FAA’s intent is to evaluate flight simulators and FTDs as objectively as possible. Pilot
acceptance, however, is also an important consideration. Therefore, the flight simulator or FTD will
be subjected to validation tests listed in the relevant ACs. These tests include a qualitative assess-
ment by an FAA pilot who is qualified in the respective airplane or set of airplanes. Validation tests
are used to objectively compare flight simulator or FTD data and airplane data (or another approved
reference data) to assure that they agree within a specified tolerance. Functions tests provide a basis
for evaluation of the flight simulator or FTD capability to perform over a typical training period and
to verify correct operation of the controls, instruments, and systems.
The above subsections should suffice as an introduction to the FARs and ACs as they apply to
defining flight simulators and FTDs, as well as to the concept of “levels” of flight simulators and

72838.indb
Human Factors266in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:52 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 267

FTDs. When we deal with LOFT later in this chapter, it will be LOFT as done in a Level D flight
simulator.

SFAR 58 and AQP

History
In 1975, the FAA began to deal with two issues: hardware requirements needed for total flight
simulation and the redesign of training programs to deal with increasingly complex human factors
problems. At the urging of the air transportation industry, the FAA addressed the hardware issue
first. This effort culminated in 1980 in the development of the Advanced Simulation Program. Since
then, the FAA has continued to pursue approaches for the redesign of training programs to increase
the benefits of advance simulation and to deal with the increasing complexity of cockpit human
factors. A joint government–industry task force was formed on flightcrew performance issues. On
September 10, 1987, the task force met at the Air Transport Association’s headquarters to identify
and discuss flightcrew member performance issues. Working groups in three major areas were
formed, and the recommendations to the joint task force were presented to the FAA administrator.
Some of the substantive recommendations to the FAA administrator from the flightcrew member
training group were the following:

a. Provide for a Special Federal Aviation Regulation (SFAR) and Advisory Circular to permit
development of innovative training (SFAR 58)
b. Require all training to be accomplished through a certificate holder’s training program
c. Provide for approval of training programs based on course content and training aids rather
than specified programmed hours (SFAR 58)
d. Require Cockpit Resource Management (CRM; 121.404, SFAR 58) training and encour-
age greater use of line-oriented flight training (LOFT)

SFAR/AQP: Overview and Synopsis

In this subsection, we will show how the relevant FAR (SFAR 58) on AQP came into being, the
portions of it that directly impact the use of flight simulator, and finally, why and how SFAR 58 and
the accompanying extensive AC have changed civil pilot training (the most significant change being
the enhanced role of flight simulator and LOFT). We will now present a brief look at the aspects of
SFAR 58 that pertain to training and to the use of flight simulators and other training devices.
Note: Any Special FAR (SFAR) expires within 5 years unless extended or made into an FAR.
In the case of SFAR 58, it would have expired in late 1995, but has been extended until October 2,
2005. It is interesting to note that the original AQP AC accompanying SFAR 58 was published in
Copyright © 2008. Taylor & Francis Group. All rights reserved.

1990. It has been updated and is in the process of finalization to be reissued in its newest version.
It was expected that this would occur in early 2004, or some 14 years since the original AC was
published.
In response to the recommendations from the joint government–industry task force and from
the National Transportation Safety Board (NTSB), the FAA put forward SFAR 58, Advanced Qual-
ification Program (AQP), in October 1990. AQP was also established to permit a greater degree of
regulatory flexibility in the approval of innovative pilot training programs. Based on a documented
analysis of operational requirements, an airline (FAA certificate-holder) under AQP may propose to
depart from traditional training practices and requirements for pilot/crew with respect to what, how,
when, and where training and testing is conducted. This is subject to FAA approval of the specific
content of each proposed program. SFAR 58 requires that all departures from traditional regulatory
requirements be documented and based upon an approved continuing data collection process suffi-
cient to establish at least an equivalent level of safety. AQP provides a systematic basis for matching

72838.indb
Human Factors267in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:52 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 268 Human Factors in Simulation and Training

technology to training requirements and for approving a training program with content based on
relevance to operational performance.

SFAR 58
SFAR 58 provides for approval of an alternate method, AQP, for qualifying, training, certifying, and
otherwise ensuring competency of crew members, aircraft dispatchers, other operations personnel,
instructors, and evaluators who are required to be trained or qualified under parts 121 and 135 of
the FAR or under this SFAR. For pilots in command, seconds in command, and flight engineers,
a proficiency evaluation—a portion of which may be conducted in an aircraft, flight simulator, or
flight training device as approved in the certificate holder’s curriculum—must be completed during
each evaluation period.
Each AQP qualification and continuing qualification curriculum must include approved train-
ing on and evaluation of skills and proficiency of each person being trained under an AQP to use
their cockpit resource management skills and their technical (piloting or other) skills in an actual or
simulated operational scenario. (The integrated assessment of CRM and technical flight skills will
be discussed later.) For flightcrew members this training and evaluation must be conducted in an
approved flight training device or flight simulator.
A person enrolled in an AQP is eligible to receive a commercial or airline transport pilot, flight
engineer, or aircraft dispatcher certificate or appropriate rating based on the successful completion
of training and evaluation events accomplished under that program if the applicant shows compe-
tence in required technical knowledge and skills (e.g., piloting) and cockpit resource management
knowledge and skills in scenarios that test both types of knowledge and skills together. (Note: There
are other requirements, but, as said, we are focusing on the flight simulator in AQP.)
As has been said, any flight simulator or FTD that will be used in an AQP for one of the follow-
ing purposes must be evaluated by the FAA administrator for assignment of a flight training device
or flight simulator qualification level:

(i) Required evaluation of individual or crew proficiency

(ii) Training activities that determine if an individual or crew is ready for a proficiency
evaluation
(iii) Activities used to meet requirements for recent experience
(iv) Line operational simulations (LOS, and to include LOFT)

AQP, LOS/LOFT, and Simulators

The capabilities and use of simulators and other computer-based training devices in training and
qualifications activities have changed dramatically. SFAR 58 and AC 120-AQP allow certificate
Copyright © 2008. Taylor & Francis Group. All rights reserved.

holders that are subject to the training and evaluation requirements of Part 121 and Part 135 to
develop innovative training and qualification programs that incorporate the most recent advances
in training methods and techniques. SFAR 58 and the AC also apply to training centers under Part
142, which intend to provide training for eligible certificate holders. AQP emphasizes crew-oriented
training and evaluation. These training and evaluation applications are now grouped under the
general term of line operational simulations (LOS), including line-oriented flight training (LOFT),
special purpose operational training (SPOT), and line operational evaluation (LOE). Due to the role
of crew resource management (CRM) issues in fatal accidents, it has become evident that LOS is the
most appropriate environment to train and evaluate both technical and CRM skills. Consequently, a
structured LOS design process is necessary to specify and integrate the required CRM and techni-
cal skills into line-oriented LOS scenarios. These should provide the opportunity for training or
evaluation, as appropriate, in accordance with approved AQP qualification standards. All of the
above can be done in an FAA-approved flight simulator.

72838.indb
Human Factors268in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:53 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 269

Line-Oriented Flight Training (LOFT)

Background
LOFT emphasizes an orientation on events that could be encountered in line operations (“flying the
line”). Thus, mission realism—making the LOFT session correspond as closely as possible to event
sets that could or would be encountered in flying one or more point A to point B legs—becomes
the major driver in LOFT design. In other words, events that make up a LOFT scenario should pass
the test of mission realism where it is reasonable to assume that this “could” happen in the real
world. The use of flight training devices and flight simulators has become increasingly important
in training flightcrew members. As the level of sophistication in simulators increased, air carriers
have come to rely on simulators for part or all of their flight training programs. Since the mid-1970s,
some FAR Part 121 and Part 135 operators have implemented alternative simulator training (now
LOFT) to train crew members. LOFT is training in a simulator with a complete crew using repre-
sentative flight segments that contain normal, abnormal, and emergency procedures that may be
expected in line operations.
This FAA AC specifies the multiple types of line operational simulations, of which LOFT is
one. The AC also specifies the types of LOFT and line operational evaluation (LOE). In this AC,
the FAA provides guidelines for LOFT content, LOFT use, LOE use, and LOFT/LOS instructor
qualifications. We will briefly show some relevant portions of this AC because LOFT and LOS are
done in a flight simulator, and because LOFT is the vital venue for pilot training and evaluation.
(Excerpt from FAA AC 120-35B 58, Line Operational Simulators/LOS, with our usual
caveat.)

LOFT is a useful training method because it gives crewmembers the opportunity to practice line
operations (e.g., maneuvers, operating skills, systems operations, and the operator’s procedures) with
a full crew in a realistic environment. Crewmembers learn to handle a variety of scripted real-time
scenarios, which include routine, abnormal, and emergency situations. They also learn and practice
cockpit resource management skills, including crew coordination, judgment, decision-making, and
communication skills. The overall objective of LOFT is to improve total flightcrew performance,
thereby preventing incidents and accident during operational flying.
The types of LOFT are:

1. Qualification LOFT—An approved flight simulator course of LOFT to facilitate transition from
training using flight simulation to operational flying. Qualification LOFT meets other require-
ments of FAR Part 121, Appendix H.
2. Recurrent LOFT—An approved flight simulator course of LOFT which may be used to meet
[yearly] recurrent flight training requirements and to substitute for alternate proficiency
checks.
3. Line Operational Evaluation (LOE)—An evaluation of crewmembers and crews in a flight train-
Copyright © 2008. Taylor & Francis Group. All rights reserved.

ing device or flight simulator during real-time Line Operational Simulations.

LOE is primarily designed for crewmember evaluation under an Advanced Qualification Program
(AQP). LOE is conducted in a flight simulator or flight training device and is designed to check for
both individual and crew competence. [Authors: Such competencies should be demonstrated in a mis-
sion-realistic environment.] LOE may also be used to evaluate a specific training objective. Operators
conducting LOE may be approved to use any level of flight simulator or flight training device, depend-
ing on the objective of the evaluation and the capability of the device. The level of the flight simulator of
flight training device required to support evaluation in LOE will depend upon the evaluation objectives
and the device’s capability to support the objectives.
Special Purpose Operational Training (SPOT) is an approved course of operationally oriented flight
training, conducted in a flight simulator or flight training device, which may be used to learn, practice,
and accomplish specific training objectives, for example, training in variant aircraft or special aircraft
equipment.

72838.indb
Human Factors269in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:53 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 270 Human Factors in Simulation and Training

LOFT is “no-jeopardy” training, that is, the instructor does not issue a passing or failing grade
to a participating crew member. As a LOFT scenario progresses, it is allowed to continue without
interruption so crew members may learn by experiencing the results of their decisions. Decisions
that produce unwanted results do not indicate a training failure, but serve as a learning experi-
ence. If the LOFT instructor identifies crew member performance deficiencies, additional training
or instruction will be provided. This training or instruction may be in any form, including addi-
tional LOFT. Before the crew member may return to line operations, the performance deficiencies
will be corrected and the instructor will document the training as satisfactorily completed.
The “no-jeopardy” concept allows crew members to use their full resources and creativity without
instructor interference. At the end of a LOFT session and after debriefing, the instructor certifies
that the training has been completed. (We will return to jeopardy versus nonjeopardy in LOFT later;
it has both a history and problematic aspects.)
To iterate: Each AQP qualification and continuing qualification curriculum must include
approved training on and evaluation of skills and proficiency of each person being trained under an
AQP to use their cockpit resource management skills and their technical (piloting or other) skills
in an actual or simulated operations scenario. For flightcrew members this training and evaluation
must be conducted in an approved flight training device or flight simulator.
The reader may feel, at this point, that what has been presented has been an overabundance of
FAA definitions, regulations, policies and guidance. This is only somewhat true and if the reading
of what has come before may have been somewhat dry and/or tedious, a point must be made again.
All of civil aviation’s activities come under the purview of the FAA. It is not possible to completely
or clearly understand the role and functions of flight simulation (whether in flight simulator or FTD)
in civil aviation without the information so far presented.

Maximizing LOFT: The Mission Performance Model

and the Operational Decision Making Paradigm
LOFT: Current and Future
We have described the initial development of LOFT, and its current form and content. We have
stated that LOFT is the major training and check tool in an AQP. LOFT and LOE, as performed in
the flight simulator, simply put, are both the optimal training/testing environment and the “court of
last resort,” as it were. Upon successful completion of LOFT/LOE, the pilot/crew has earned new
ratings or certifications or are “good to go” for another year.
However, the current LOFTs and LOEs need to be strengthened for exactly the reasons cited
above; they are the best and safest methods for cutting edge, realistic training and evaluation, and
they provide a final stamp of approval in an AQP—as well as a more traditional Part 121-based
training program. We have set the stage to present how our earlier statements about the tremendous
Copyright © 2008. Taylor & Francis Group. All rights reserved.

potential and existing use-value of LOFT can be merged and realized via the MPM and the ODM
models.

Risk Identification and Management: Training and Evaluation

with MPM and ODM Paradigm

The end result of all civic pilot training should be to prepare a pilot to identify, assess, and man-
age risk. The primary role of the pilot as a risk manager has been emphasized multiple times over
the past 10 years by the authors (Lofaro, and Smith, 1993, 1998, 1999, 2000, 2001, 2003). LOFT
is simply the preeminent tool, as well as test situation, for training and evaluating civil captains/
crews. Over the years, two major models have been developed by which LOFTs can be designed
and crew performance enhanced as well as evaluated. The first is the Mission Performance Model
(MPM) as developed by Captains Kevin Smith and William Hamman of United Airlines, with

72838.indb
Human Factors270in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:54 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 271

some input from Jan DeMuth and Ron Lofaro of the FAA. The MPM came from the recognition
that the CRM skills must be integrated with a corresponding set of technical skills (flight control
skills) in an interactive matrix in order to fully evaluate overall crew proficiency. Further, such
an integrated CRM approach would serve as a training tool—in LOFT design and in specifying
where the CRM/flight control skill linkages existed. An approach to integrated CRM, along with
both human factors and flight control/technical skill evaluation scales, was partially developed
during an FAA-hosted workshop in 1992. Dr. Lofaro was the designer and facilitator of this work-
shop and Smith, along with several training captains from NW, DL, United Airlines, the chief
pilot for Boeing, 2 FAA personnel, an aviation psychologist for the USN Postgraduate School,
and a research psychologist under contract to the FAA were the participants. The results of that
workshop are in Report DOT/FAA/RD-92/5: Workshop on Integrated Crew Resource Manage-
ment (Lofaro 1992).
The integration, and assessment, of crew resource management (CRM) and flight control
skills had received considerable attention—and a fair share of concern and skepticism—in the
1980s and early 1990s. As one response, the ATA formed a joint air carrier/FAA/academic work-
ing group to deal with this and other CRM issues in 1990; both Smith and Lofaro were on that
group. Dr. Robert Helmreich, in conjunction with several major air carriers, developed a complete
set of flightcrew CRM performance markers (he termed them “CRM behavioral markers”) with.
behaviorally-anchored rating scales. In a NASA/FAA/University of Texas project, Helmreich
worked with several air carriers on research that involved the use of these markers in LOFT.
In 1991, Smith and DeMuth developed an initial set of performance markers for the technical/
flight control skills. Both the CRM and the technical sets of markers were used in the next step of
CRM integration: the attempt at developing an analytic paradigm. Smith created the framework for
a model that demonstrated that the CRM human factors skills and the technical/flight control skills
are interrelated, interdependent, and often simultaneous in execution—that, for safe and efficient
flight, CRM can sometimes be integral to flight control, and vice versa. This model is called the
Mission Performance Model (MPM). Smith worked with Hamman and others to develop exemplars
of the application of the MPM to actual flight maneuvers, such as an engine-out at V1, with a turn
procedure required by the terrain.

Mission Performance Model

The model is based on these concepts:

1. Flying is an integrated, mission-orientated activity and must be evaluated as such.

2. The crew’s performance is not adequately captured by totaling the sum of the component
tasks/subtasks/elements. The focus must be on crew function—usually at the task and
critical subtask levels.
Copyright © 2008. Taylor & Francis Group. All rights reserved.

3. Flight proficiency skills/knowledge are interwoven, interdependent, and necessar-

ily interact with the CRM skills/knowledge differentially across tasks and conditions.
These interactions can be identified/specified by a matrix-type crew mission perfor-
mance model using the tasks, which comprise a mission/flight leg. (This is what we term
integrated CRM.)
4. The model can capture these interactions and can be sensitive to changes in both task and
mission—for example, show that, for different tasks and conditions, the technical/flight
proficiency skills, the CRM skills, and their interactions will vary. This is an indication
that the model has a measure of discriminatory power or “sensitivity” to changes in task
and conditions.
5. Helmreich’s behavioral markers can adequately delineate CRM skills and provide one
basis for the (flightcrew) Mission Performance Model, as can the technical markers cap-
ture the flight control skills and form the other MPM basis (see below).

72838.indb
Human Factors271in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:54 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 272 Human Factors in Simulation and Training

Finally, the bases for the technical proficiency evaluation currently exist in a behavioral marker-
type format with scales. Both the marker and their scales can be validated/modified for evaluation
of all these proficiencies, which will be called “crew performance markers—technical factors.”
This arena focuses on the crew as a unit and how well they discharge the technical aspects of the
mission. It specifically addresses precision maneuvers across these areas:

1. Flight maneuvers and attitude control

2. Propulsion/lift/drag control
3. System operations
4. Malfunction warning and reconfiguration
5. Energy management

Another rationale for the MPM, and later, ODM, is that pilot/crew performance has often been
seen as series of discrete tasks, where each task was further decomposed to reveal a set of subtasks
combined with the requisite knowledge and skills necessary for subtask completion. For many
applications, such as aircrew training, this produces a large collection of task, knowledge, and
skill data. In most traditional pilot or crew training programs, these are taught individually as iso-
lated knowledge components. Consequently, the trainee is left with the responsibility of combining
these isolated knowledge components into integrated wholes (Merrill, 1989). However, the linear
decomposition of individual tasks does not address integrated functioning nor does it reveal how
tightly coupled teams (flightcrews) perform, thus an analytical process other than the traditional
task analysis approach is considered necessary. Therefore, the MPM uses the functional modeling
approach.
The Mission Performance Model has embedded within it the concept of functions. It is proposed
that the model, as constructed, represents all significant functions necessary for the successful
completion of an air transport mission. This model views crew performance as consisting of system
level functions that represent the mechanisms used to perform a mission activity. The importance
of a model that is founded on a set of systems level functions cannot be overstated. Moreover, the
model delineates crew performance at a level of abstraction that is significantly different than the
current descriptions of individual performance.
The MPM consists of a set of functions that can be activated by inserting an instance/.
example—in other words, asking the function to specify/describe a particular activity or situation
in the mission. If a particular function such as workload management was asked to “spin out” the
components of a particular mission activity, such as takeoff with an engine failure at V1, then the
function should be able to organize, sequence, distribute, and coordinate key crew actions so that a
successful outcome could be assured.
This workload management function, then, can be viewed as a generic performance state-
ment that
Copyright © 2008. Taylor & Francis Group. All rights reserved.

a. Can be applied to many mission activities/situations, and

b. Can be activated for the application to, and specification of, any one of these activities/
situations.

The Mission Performance Model specifies the components of flightcrew “effectiveness” (effective
performance). That the model represents effectiveness is important to understand since, if the crew
is really engaging in the set of functions that are both germane and linked to the problem at hand,
and if these functions are the prerequisites for a successful outcome, then effectiveness has been
demonstrated.
Similarly, the model is prescriptive; it prescribes what needs to be accomplished for the crew to
perform effectively. For example, we can specify, during the LOFT design process, what are very
likely to be the necessary crew behaviors.

72838.indb
Human Factors272in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:55 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 273

In summary, in the MPM, human factors as well as technical performance clusters are specified
along with the applicable markers under each cluster. For example, under workload management
and situational awareness, key markers include preparation, planning, vigilance, workload distribu-
tion, and distraction avoidance. Similarly, under the cluster titled “propulsion/lift/drag control,” the
key makers include instrument interpretation, energy management, power control, lift control, and
drag control. When all these markers are combined into a matrix array with their various categories,
the MPM emerges.

From CRM/MPM to ODM

Upon completion of the 1992 Integrated CRM Workshop, a new set of issues and concerns became
apparent to Smith and Lofaro. The Integrated CRM concept and the MPM were well received by the
workshop participants. However, due to many factors—such as a lack of FAA interest for follow-on
efforts and a CRM “establishment” that was not open to taking CRM to either another level or in
new directions, along with the jeopardy issue—it was clear that Integrated CRM and the MPM had
become dead issues. Of much more import was the realization that CRM was not the human factors
silver bullet. Captain Hal Sprogis asked, “Is the Aviation Industry Experiencing CRM Failure?”
(Sprogis, 1997). Captain Daniel Maurino had written “Crew Resource Management: A Time for
Reflection” (Maurino, 1999). Both indicate that we may have expected too much from CRM; that
the relationship between CRM and safety, which was and is the prime rationale offered for teaching
CRM, has not been proven; that CRM is a process, not an outcome, and certain efforts to assess out-
comes (i.e., individual performance) may be misguided. In July of 1996, American Airlines decided
to totally revamp their CRM training program. Its reason was that flightcrews had valid objections
to, and concerns about, CRM: “CRM is too often viewed as a number of interpersonal issues that
simply do not define the problems that we face in aviation … . CRM training will most likely always
be defined and suffer in terms of the first generation of courses … which were seen as ‘touchy-feely,’
‘getting along,’ and ‘managing human relations or resolving personality conflicts’ rather than deal-
ing with truly important concerns” (Ewell and Chidester, 1996). American’s new focus is on pre-
paring flightcrews for the daily challenges of normal and abnormal operations encountered flying
the line. Delta Airlines, in the same timeframe, revamped its “CRM for New Captains” course and
now calls it “In Command.” As with American, Delta emphasized leadership, responsibility, and
performance. So, in 1996–97, we see these two major carriers eschewing overemphasis on commu-
nication and interpersonal relations in their CRM training. Lastly, United Airlines’ version of CRM
was and is, called C-L-R, where the C is for “command” and the L is “leadership,” indicating that
United wanted to bypass the interpersonal with C-L-R and move on to the performance issues. Yet,
even United changed aspects of its CRM in 1997.
Further, “common wisdom” was that pilots made good decisions easily and almost naturally,
aided by (some) increase in experience. The facile assumption that additional experience will teach
Copyright © 2008. Taylor & Francis Group. All rights reserved.

pilots to make better decisions has proven to be a dangerous fallacy. Experience can be a nasty
teacher, often giving the test before, or without, giving the lessons and materials needed for the test.
Experience can also reinforce poor decisions and behaviors that seemingly “worked” in the past
(the “not your day to die” phenomenon). There was also the commonly accepted view that decision
making is but one of the components of CRM. This was, and is, a gross error.
CRM, with its emphases on communications and team function, is but one enabler of good
decisions. As such, it is a part of decision making, not vice versa. CRM is, simply put, an enabler
of decision making. Decision making is the primary tool to be used by the pilot and crew with their
primary functions: risk identification and risk reduction. In short, risk management.
It also became apparent that aeronautical decision making was greatly different that decision
making on the ground, and that a new paradigm was necessary that both articulated the differences
and had a new set of decision making (DM) techniques specific to what pilots and crew encounter.
Another realization was on the primacy of LOFT in pilot training. As one result of this Hamman,

72838.indb
Human Factors273in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:55 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 274 Human Factors in Simulation and Training

Seamster, Lofaro and Smith wrote an interrelated set of papers on LOFT design and delivery, that
later formed a session at the 1993 International Symposium of Aviation Psychology biennial meet-
ing in Columbus, Ohio.
As another result, Lofaro designed and held a FAA/Industry/DoD/Academe Workshop in
Denver (1992) which had some of the CRM workshop participants and added others from the deci-
sion-making world. The two-volume FAA report on this workshop (DOT/FAA/RD-92, Vols. I,
II; Lofaro and Adams) initiated the efforts for what has become the operational decision making
(ODM) model of Smith and Lofaro (Lofaro and Smith, 2000, 2001, 2003)

THE ODM
Operational Decision Making: New Window of Opportunity
In the new millennium, there seems to be a heightened level of recognition of both the role and the
importance of decision making in aviation safety. In July of 2000, Aviation Week and Space Technol-
ogy printed four somewhat interrelated articles on pilot/crew errors and decision making in aviation
safety. These articles presented audit information, research, and analyses along with current “fixes.”
Some findings include flightcrews ignoring increased evidence that the original flight plan was no
longer appropriate; tactical DM (what we could call operational DM) errors, which are the second
most prevalent crew errors in crew-involved accidents; and pilot perception of risk, a key factor in
safety. Several other vital points were that (1) it is a fallacy that crew error can be eliminated via
sufficient vigilance, and (2) there is a need to emphasize managing the consequences of error. The
proposed “fixes” that were offered encompass risk and error management training as well as new
decision-making training. All in all, these articles showed us that many of the components of ODM
we have developed and published since 1992 were on target. However, we believe that there were
critical components missing, as well as a coherent structure, or model embedding the components.
In addition to the above problems, we sought to address such things as the “common wisdom”
referred to above and some training issues that follow.

Introduction to the ODM Paradigm

Over the past 12 years, we (Smith and Lofaro) have zeroed in on decision making in the operational
air carrier environment and developed a paradigm called operational decision making (ODM). Dur-
ing that period, we have introduced and published on such concepts as event sets in LOFT (1993);
the pilot as risk manager (1992, 1993, 1998, 1999, 2000, 2001); the primacy of decision making over
CRM (1998, 2000, 2001). The focus of ODM is on enabling the pilot’s primary role, that of risk
manager, by using the model; this essentially redefined the role and activities of the pilot. Safety
becomes the operational aspect of successful mission completion through risk management; ODM
is the key to successful risk management and good decision-making skills are the primary tool in a
Copyright © 2008. Taylor & Francis Group. All rights reserved.

pilot’s safety arsenal. We see decision making as the key pilot activity. How can a captain command
and lead if he or she is not prepared (trained and practiced) to make the best decision possible? This
must be the focus for developing, then training, captains to use an operational decision-making
process. Any ODM process must function in real-time, and the ODM training must enable a captain
and crew to both identify and mange risk. The need for operationally oriented decision-making
training is vital for pilots and crews of major carriers, with their highly complex, automated craft
and their responsibility for hundreds of lives each time they fly.
Further, in developing our ODM model, we sought to address what we see as a shortfall in
current aircrew training. Air carrier training today is built around (1) a sequence of discrete events
and/or conditions and whether or not these are, separately, within what is termed “legal limits” ( as
per FARs); and (2) the pilot’s responses to these events. However, it seems clear to us that all too
often there exist interactions and interplay of seemingly discrete events and conditions. Although
any one or all of a group of events and conditions might be within limits, the interactive resultant

72838.indb
Human Factors274in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:56 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 275

(cumulative effect) of them may place the aircraft and mission at risk. We are not aware of air car-
rier training that uses this perspective. We believe that our ODM model directly addresses this gap
in aircrew training.
In summary, ODM deals with the complex issues and methods for the pilot/crew identification
and evaluation of risk factors as the necessary skills for managing such risk and thereby achieving
mission success. The ODM should be embedded in either qualification on recurrent training as the
apex of CRM and should be taught immediately after CRM.

Components of the ODM Model

Integrated throughout the model is the pilot’s role as risk manager. The other components are:

1. The operational envelope

2. Situational knowledge/risk location
3. The critical mission impact areas; the critical mission factors that comprise the impact
areas/risk location
4. The rising risk continuum/risk location cumulative effect (a concept whose use is embed-
ded throughout all the above components)

It must be strongly noted that all of these components are so intertwined that any separation or
sequencing of them is artificial, because it will become apparent that to speak of any one component
brings in the others. Actually, there is no sequence per se. Rather, there is a constant interplay and
branching across the components; perhaps a continuously rotating circle best describes the process
the pilot is involved in. However, for the purposes of presentation, we will treat each one separately.

Risk Management: Risk, Rising Risk, and Flying the Line

Mission completion and risk management involve completing the mission of flying the public
because of its economic benefits, but in a way that does not place people or equipment at undue risk;
these should be the goals of any airline pilot. By risk, we mean specific risk or danger to the air-
craft, passengers and crew, and the corporation. There are three levels of risk in the paradigm. High
risk is defined as the likelihood or high probability of injury, damage, or death. Moderate risk, if
left unchecked, could continue to rise and/or likely result in significant flight trajectory deviations.
Low risk, finally, is a normal situation where routine, normal procedures are sufficient. Rising risk
refers to the facts that (1) if some, or any, problems went unnoticed and/or (2) the decisions were not
accurate, timely, and appropriate, then (3) the risk to the successful completion of the flight could
rise to the point where the flight is truly endangered. We will return to these later. If the level of
risk rises, effective risk reduction strategies need to be employed to keep that risk within manage-
Copyright © 2008. Taylor & Francis Group. All rights reserved.

able limits. Risk management is critical to the retention of the flying public customer base and the
long-term viability of the industry. Risk management is the key operational activity that works hand
in glove with mission execution skills. Central to a completed and safe mission is the pilot as risk
manager, an operational responsibility that undergirds the model.

Risk Management: The Operational Envelope, Risk Location, and Cumulative Effect
We will work from the center of the model outward, then return to the inside. The first—and last—
component in the spiral-like process is a graphical representation of the boundary conditions in a
mission. This is the pilot’s “world,” and we term it the operational envelope, as does the military.
It gives, to use a hackneyed phrase, the big picture; the “ops envelope” is the context in which all
the ODM paradigm’s components are activated. The operational envelope gives a rather straight-
forward way to grasp the pilot’s task universe. The ability to locate oneself within the ops envelope
results from the pilot’s situation knowledge and ability to do accurate risk location. It would seem

72838.indb
Human Factors275in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:57 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 276 Human Factors in Simulation and Training

Airspeed, Attitude, Workload

System and Human Limitations

Adverse Mission
Conditions Critical
Alerts
&
Normal Conditions
Adverse Warnings
Normal SOP
Wind

Clutter
GPWS
Icing
Restricted Visibility Stall
Terrain Warning

TCAS
Non-Normal
Conditions Less Than 1000-3 Radar

Figure 14.1 Operational envelope.

that civil aviation has never taken this larger view of the envelope because it has focused on discrete
data points instead of on the big picture. Even today, we find that Airbus and Boeing are working
on software for flight control computers that are aimed at keeping the aircraft from exceeding a
predetermined flight envelope (Aviation Week and Space Technology, August 28, 2000). However,
these efforts focus on alarms and/or software to stop overstress, possibly dangerous pitch angles
and over banking; in essence, these deal with a small, specific set of discrete flight control errors,
not decision making.
The ops envelope can be visualized as a four-sided figure, composed of sides that intersect at
right angles to each other. These sides—“boundaries”—are adverse conditions; restricted visibility;
mission critical alerts and warnings; and system and human limitations, as shown in Figure 14.1.
However, what we show in this figure is a flat cross section. The actual envelope is three-dimensional
as the plane cannot only fly any direction on the compass rose, but can also climb and descend.
Although this seems obvious, we remind the reader not to lose sight of the fact that we are using a
linear representation for a 3-D world.
Figure 14.1 illustrates the operational envelope and shows some sample conditions that are
embedded in each boundary/side. The first risk management aspect of the operational envelope is
that, at the very center, there is an area of normal flight conditions. Within this inner area, normal
SOPs will suffice. However, a single event such as freezing rain, or event sets, such as contamina-
Copyright © 2008. Taylor & Francis Group. All rights reserved.

tion plus strong crosswinds, encountered during flight may drive a pilot and crew out of that inner
area and toward the edges/boundaries. If that happens, there may still be guidelines on what to do.
But, the second risk management aspect is that, although each event or factor may be able to be han-
dled by itself, and keep the aircraft inside the edges/boundaries of the envelope, how do we handle
the combined or cumulative effect, that is the result of multiple events and factors? If the cumula-
tive resultant of combined factors in the event set drives us toward any corner, we first encounter
what we termed a “gray zone” where we are still within the envelope, but close to a corner. If near a
corner, in a gray zone, there exists no SOPs or procedures. Certainly, cumulative effects will effect
one’s position in the ops envelope and, thereby, impact risk. We will return to this later.

Risk Management: Critical Mission Impact Areas and Their Components

Critical to mission success is the ability to operate in adverse conditions while simultaneously exe-
cuting the mission plan and managing risk. The need to have the capability to operate under adverse

72838.indb
Human Factors276in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:58 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 277

conditions and thus provide reliable transportation is self-evident. But this leaves unanswered what
events/factors cause a (rising) risk: What are some of the “risk drivers”? We must first look to iden-
tifying, defining, and understanding the mission-critical impact areas (MCIA) that engender risk.
A mission impact area, if encountered, denotes some level of risk and, if action is not taken, will
cause the risk to rise.

Boundary Conditions as Mission Critical Impact Areas

We posit the boundaries/sides of the ops envelope function as the critical mission impact areas.
Table 14.1 shows the boundary conditions of the ops envelope, with their subareas and critical mis-
sion components.
Note: As one rule of thumb, any factor encountered on a mission will cause the risk to rise; two
factors generally will result in moderate risk; an interplay of three or more factors usually results in
a high-moderate to high-risk situation.

Risk Management: Situation Knowledge and Risk Location

The continuous task of the pilot is risk identification by using situation knowledge and consequently,
accomplishing (ever-changing) risk location. Accurate and timely risk location is the only way to
achieve both an accurate and timely action response to risk. Situation knowledge and risk location
are interrelated concepts. Situation knowledge is that part of the ODM structure that consists of the
continually changing set of elements (knowledge bits) comprising the pilot’s awareness of (1) the area
of the ops envelope in which the captain believes the aircraft is located and (2) which of the critical
components of the ops envelope boundaries are in play. The result of these components “tells” the
pilot what is the cumulative effect of the critical factors that are in play. By using the result of a
cumulative effect, the pilot can relocate the aircraft in the operational envelope. This enables the
pilot to use the rising risk continuum as a decision tool for action response. As said, accurate risk
location is the key, when in a (rising) risk situation, to making the optimal selection of a course of
action—for example an action response that is an alternative to the original mission plan; we will
return to this later.

Risk Management: Rising Risk, Risk Location, and Response Action

We view rising risk as a teaching concept and the decision-making tool that enables the crew
action responses needed for optimum risk management. The teaching aspect of rising risk is in
engendering the awareness that, as issues and problems arise, the risk will rise. As risk will rise
under virtually any set of adverse conditions, risk management demands that the crew execute, in
a timely fashion, specific mitigation procedures to prevent “risk migration to the right” into the
moderate and high-risk areas. Rising risk also is the decision tool for response actions, based on
Copyright © 2008. Taylor & Francis Group. All rights reserved.

the rising risk continuum. If risk is low, the original plan can be executed to completion. When
the risk rises to a moderate level, modifications to the original plan must be implemented in order
to maintain an acceptable “location” in the risk dimension. When the risk rises further to some
critical threshold, the original flight plan needs to be discontinued and the mission is aborted. In
summary, if the risk rises, the captain, working with dispatch and other crew members, either (1)
continues with the mission as originally planned for low risk situations, (2) modifies the mission
plans as needed in moderate risk situations, or (3) abandons the mission altogether in high-risk
situations.
However, it can be seen that the pilot must, in order to have an accurate perception of risk
location, have an awareness of any cumulative effect on the plane and any adverse factors that are
in play. In short, the pilot’s perception of the risk must be consonant with the reality of the plane’s
location in the ops envelope. We now return to cumulative effects and their impact on risk location
in the ops envelope.

72838.indb
Human Factors277in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:59 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 278 Human Factors in Simulation and Training

         Table 14.1
Boundary Conditions of the Operational Envelope
Adverse Conditions Mission Critical Alerts and Warnings
Adverse weather Stall warning
Slippery Predictive windshear
Clutter Actual windshear
Contamination Predictive terrain
Icing aloft Actual terrain
Freezing precipitation TCAS RA
Volcanic ash Improper configuration
Convective activity PRM breakaway
Adverse wind Radar Doppler return
Headwind Convective return w/ hook
Crosswind Wing contamination
Tailwind Any severe conditions
Severe gust Upset
Windshear Wake turbulence
Nonnormal operations System and Human Limitations
Nonprecision approach Performance limited ops
ICAO takeoff Clutter
Circling approach Weight/fuel
Day-only restrictions Flaps
Cold temperatures Packs
Extreme latitudes Delayed VR
Raw data Cruise/speed altitude
Unfamiliar Landing
Engine out operations Missed approach
Takeoff alternate MEL
T-procedure Human limitations
V-1 cut Memory saturation
V-2 cut Task demand and overload
EO approach Channel saturation
EO landing Information overload
EO missed approach Fatigue
Copyright © 2008. Taylor & Francis Group. All rights reserved.

ETOPS divert Activity prioritization

Risk awareness
Restricted Visibility
Multitasking
Below 1000/3
Task interference
Below circling minimums
Below CAT 1
Below CAT 2
Below CAT 3
Takeoff below landing minimums
Blowing snow
Blowing sand
Patchy fog
Night mountainous

72838.indb
Human Factors278in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:22:59 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 279

A Distinction: Cumulative Effect Resultant versus Additive Effect Resultant

We have discussed the cumulative effect that results from the interaction of factors that come from
any two contiguous sides of the ops envelope. In this, we have relied, albeit loosely, on vector/ten-
sor mathematics; our cumulative effect vector is somewhat analogous to the resultant vector that
can be computed from two or more forces at right angles to each other. The key points here are:
(1) The cumulative effect resultant (CER) has a new and different direction than either/any of the
forces that comprise it, (2) the CER drives the plane toward a corner, and (3) the CER changes
the shape and position of the corner the plane is being driven toward. As the cumulative resultant
of combined factors in the event set drives us toward a corner, we first encounter what we termed
a “gray zone” where we are still within the envelope, but close to a corner. If near a corner, in a
gray zone, there exists no SOPs or procedures. The same is true if forced out of the envelope (see
Figure 14.2).
As said, the CER of an event set, composed of critical mission (and boundary) factors can drive
a pilot toward one of the four corners (pun intended, as you are really “cornered”). The closer to a
corner, the more the risk has risen. However, the corner of the ops envelope has also changed shape;
it has become a triangle, with the (45°) hypotenuse of the two sides being the new boundary line.
Thus, the corner boundary lines are replaced by the new line (the hypotenuse), termed a “special
boundary.” This special boundary is closer in than the former corner so that before you would get
to the former corner, you have actually crossed this new (special) boundary line and are outside of
the envelope (see Figure 14.3). A typical set of conditions that cause a change in the position and
shape (becoming a new boundary line) of what once was a corner could be low ceilings and visibil-
ity (restricted visibility) with a reduced braking effectiveness due to a slippery runway and strong
crosswind (adverse conditions).
There is another effect that comprises more than one factor and which can act in a way to drive
the aircraft either out of the ops envelope area of normal operations or out of the envelope entirely;
we will term this the additive effect resultant (AER). The difference here is that an AER is com-
posed entirely of factors that do not give the resultant a new direction. This is to say that all the fac-
tors in play come from one and only one particular mission impact area or set of boundary factors.
These factors combine, additively, to produce an effect larger than any of the factors. However, the
AER pushes the plane toward a boundary of the envelope, rather than a corner. One simple example
would be a series of adverse conditions such as icing aloft, clutter on the way down, and slippery
runways on arrival. Pilots seem to be able to recognize that these linked factors do raise the risk
and, usually, take the necessary actions. Still, we are surprised at the level of stress engendered
when more than one factor is encountered.

Strong Crosswind
Copyright © 2008. Taylor & Francis Group. All rights reserved.

And
Blowing Snow
Adverse
Conditions
Cumulative Effect Resultant

Visibility Less
Than 3 Miles

Restricted Visibility

Figure 14.2 Cumulative CER.

72838.indb
Human Factors279in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:23:00 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 280 Human Factors in Simulation and Training

Strong Crosswind
And
Blowing Snow
Adverse
Conditions

CER

Visibility Less
Than 3 Miles

Thus, the corner

Becomes a
Special boundary
Restricted Visibility

Figure 14.3  Formation of a special boundary.

Operational Envelope and Rising Risk Continuum: Relationship

There is no simple isomorphic relationship between the aircraft’s position in the ops envelope and
its position on the rising risk continuum. However, by looking again at the ops envelope, we can
develop a loose mapping to take us from a position on the ops envelope to the risk continuum. What
follows is somewhat rough, but seems intuitively obvious. This aspect needs refinement and clarifi-
cation. Once again, the authors are working on it.
Clearly, if a plane is in the normal ops zone of the ops envelope, that is roughly equivalent to
being at low risk. The nonnormal ops zone is, then, roughly equivalent to at least moderate risk.
Penetration of any boundary, or special boundary, of the ops envelope puts the aircraft at high risk.
The gray areas, where an aircraft is nearing a corner due to a CER, we term as high-moderate,
bordering on high-risk.

Putting It All Together: Risk Location and Risk

Management in the Operational Envelope
We return to the ops envelope and the risk location within it—in other words, the big picture, or in
some current phraseology, maximal situation awareness. As was said, all the paradigm’s compo-
nents are intertwined, and the pilot is using them by a constant process of mentally rotating through
them; a process that, as it deals with three axes, is a spiral one. The pilot must always ensure that his
risk location encompasses not only the critical mission factors but the plane’s location in—or, more
Copyright © 2008. Taylor & Francis Group. All rights reserved.

dangerous—outside the ops envelope, and the risk factors in play due to that position as well. This
is what we have termed cumulative effects. By returning to the ops envelope focus, the pilot can
accurately place the plane on the rising risk continuum, then take the needed action. Risk location
in/out of the ops envelope, translated into risk location on the rising risk continuum, followed by
response actions are the keys to successful risk management.
The challenge is to use our situation knowledge to accurately develop the result of the com-
bined critical factors and locate ourselves (risk location) in or outside the ops envelope. Next, we
must translate our ops envelope location into our location on the risk continuum. Once we locate
ourselves accurately on the risk continuum, taking into account the cumulative effect of being in,
or very near, a corner, with all the factors that are in play, we can formulate and execute optimum
decisions. Again, when we encounter this phenomenon of cumulative effect and rising risk (gray
zones; new, special, boundary lines), the needed decisions cannot be made by simply following
procedures.

72838.indb
Human Factors280in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:23:01 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 281

ODM and MPM in LOFT Design, Development, and Evaluation

Introduction
We must deal with major issues before we go into LOFT design using ODM and MPM. The first is
that CRM training was first developed in the late 1970s/early 1980s after a series of disastrous and
fatal air carrier accidents—accidents where perfectly functioning planes crashed. Human-factor
errors by pilot/crew were seen as the cause of these accidents. As a result, the FAA wanted the
air carriers to implement new, human-factors-oriented training—for example, CRM. Rather than
append existing FARs to make CRM mandatory, the FAA chose another path. (Of interest here is
that in SFAR 58, the FAA decided, after many years, to make CRM mandatory in AQP training.) To
return to the situation at hand, the FAA, in order to make CRM training costs palatable to air carri-
ers, offered to waive some hours of pilot recurrency training in lieu of CRM training. As training is
a “big buck” item for air carriers, this allowed the carriers to save (not spend additional funds) and
give CRM. It also allowed the FAA to ensure the CRM training would, to a great extent, be given,
thus silencing some critics who, understandably, wanted new human factor training to counter the
rash of accidents.
However, there was a sticking point: jeopardy. Simply put: As a further inducement to carri-
ers to give CRM, the FAA and ALPA agreed that CRM would be “no jeopardy” training. When
CRM skills were evaluated in a LOFT, neither pilot nor crew could fail or be given a “down,” which
requires additional training and checking. Therefore, the evaluation of a LOFT such as the CRM,
consisted of videotaping the LOFT and a critique/debrief given to the crew upon completion of the
LOFT. The videotape is then erased.
Our view is that LOFTs built around the MPM and ODM must be evaluated as a jeopardy
LOFT session. The MPM and ODM must be evaluated as a jeopardy LOFT session. The MPM has
technical markers that encompasses actions and skills normally evaluated in a check ride (flight
simulator or actual flight), which can be failed. These include attitude management, course devia-
tion, power management, etc. In short, the full panoply of flight skills that are usually evaluated in
sessions/flights can be included, so it would seem that failure should be an option. Added to this, we
would have LOFT scenarios that enable ODM, and both pilot and crew must be allowed to succeed
or fail.
We have postulated ODM as the primary tool for training pilots and crew to do risk identifica-
tion and management—and that risk identification and management are the primary functions of a
flightcrew. The common fallacies about decision making were discussed earlier in the chapter. We add
this: One of the authors (Smith) functioned as a “line check airman” during the late 1990s. He was
checking fairly senior pilots transitioning to a large (250-plus passenger capacity), highly automated
aircraft. He was saddened (and both angry and frightened) to see the number of times pilots either did
not recognize a decision point in time to stay ahead of the power curve; did not recognize a decision
Copyright © 2008. Taylor & Francis Group. All rights reserved.

and action point at all; and made poor decisions—decisions that raised the risk for having an unsafe
and unsuccessful flight. In conversations, the authors became even more convinced of the absolute
need for ODM.
What LOFT not only lends itself to evaluation but is actually designed for evaluation? A LOFT
that uses event sets with embedded decision points carefully designed to force decisions on a risk
continuum, and a LOFT that is designed with event sets partitioned into the MPM matrix, thus
showing both CRM and flight skills. The evaluations of these carry with them the possibility of
failure, without which they are meaningless.

LOFT: ODM and MRM

In designing and developing LOFT scenarios, the basic unit, as proposed in 1993 (viz. Hamman,
Smith, Lofaro, and Seamster, 1992) is the event set. The LOFT scenario is a set of event sets
selected from real-world ops reports or made from amalgams of events and incidents as reported,

72838.indb
Human Factors281in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:23:02 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 282 Human Factors in Simulation and Training

or actually put together from the experiences of the LOFT design team. (An aside: NTSB accident
reports can also be used.) It should be clear that superior LOFT design is a team effort, and the team
should be carefully selected. The LOFT design team must consist of senior pilots with extensive
flight time in the aircraft type the LOFT scenario is being created for. These pilots should also
have experience in the air carriers’ training complex. It is also somewhat desirable to have a person
from the training department with ISD credentials, as will be shown later. One LOFT design team
member should be a flight simulator operator to ensure that the event sets selected can be replicated
in the flight simulator. Finally, it is truly preferable if one or more LOFT design team members have
been line check airmen.
Having selected the team, the next step would be to layout an overview of the LOFT mission/
flight leg(s). This overview would include the basics, such as Wx, time-of-the-year ops (i.e., winter),
and departure and destination airports, as well as alternatives. Into this skeletal framework, the
team will select the event sets for each phase of flight (takeoff, cruise, descent, landing) as well as
any pretakeoff event sets that may impact the flight leg.
The next steps are the crucial ones: Carefully select the problems that you want the flightcrew
to encounter: mechanical, system malfunctions, etc. Then, plan the sequence into which you want
to embed the problems. Remember that the overall goal is not to create the fabled “LOFT from
hell”—one which cannot be successfully flown but must result in a loss of flight control.
In both the sequencing of the event sets and the selection of the problems to be embedded in
the LOFT, the MPM and the ODM are to be used as the structural underpinnings. It is done in this
manner:

1. Upon selection of the problems and the phases of flight that these problems are to occur
in, the ODM is used to build a sequence that results in a rising risk. The decision points
are identified. (A “decision point” is a point in the flight where if no decision and resultant
action is taken, or if a wrong decision is made, the risk rises from low to moderate or from
moderate to high.) Upon identifying the decision points, the basic sequence is modified
to add the various outcomes from no decision/wrong decision; that is to say, the sequence
now contains branches that are dependent upon the decisions made/unmade. Each branch
or node will also need to have any changes in conditions and systems (again; Wx, en route
or at destination, systems malfunctions, etc.) built in.
2. Next, the MPM is integrated with ODM. The concept here is to make the consequences of
following a no decision/wrong decision model such that the risk continues until it reaches
the high level and crew action must be taken in order to regain any possibility of success-
ful flight completion. The “successful completion” may evolve ATB or diversion to the/an
alternate airfield where “success” simply means landing the plane.

This integration is a two-step process. The first involves taking the selected event sets and identi-
Copyright © 2008. Taylor & Francis Group. All rights reserved.

fying the critical tasks that are to be performed during those times. These high-level critical tasks
(e.g., V1 “cut” on takeoff) are then decomposed, using the ISD process, into the complete list of
subtasks involved. In the second step, the MPM is then used to further identify which of the critical
tasks track to which of the relevant CRM and lift control functions necessary for successful task
performance. The set of MPM functions will organize, sequence, distribute, and coordinate the
actions key to successful performance.
Looked at another way: On the V1 “cut” at takeoff, as an examplar, we find that the needed
CRM function is workload management. The MPM, with ISD decomposition, will spin out the
specifics of the critical actions and flight control skills embedded in the workload function. The
flight control tasks for this example include propulsion/lift/drag, operational integrity, and altitude
control—with such subtasks as disconnecting the auto throttle at 400' AGL, setting airspeed to xyz
knots, checking flap setting, and so on. The MPM will also spool out the crew performance mark-
ers for each subtask—both the CRM and flight technical markers. Not only that, but the functions

72838.indb
Human Factors282in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:23:03 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 283

and actions of both the pilot flying (PF) and the pilot not flying (PNF) will be clearly spelled out.
These, as said, will be spelled out at the subtask level. In fact, this is true CRM integration—the
place where both CRM and flight control actions are presented as a unified whole. However, space
and scope preclude further explication, and there is a CRM integration document (Lofaro, 1992).
Because the necessary performances are specified, the performance markers can be used not
only to track the crew’s actions but, if desired, to evaluate them. This evaluation can be done simul-
taneously using a flight simulator operator and a check airman, or done post hoc using the vid-
eotapes that are normally part of LOFT sessions. Again, scope precludes going further into the
evaluation area.
In summary, we see that the LOFT design has been driven by using the ODM to do initial event
set selection and sequence design. Then, the MPM was used to generate the task and subtask break-
outs for selected events within the event sets. The MPM further sequenced the events selected (as
an example, CRM and flight control integration was performed at the subtask level, with optional
evaluation procedures).
The LOFT design can be seen as embedding event sets into the LOFT scenario that can take the
crew and plane into the moderate and even the high risk areas of the rising risk continuum, thus:

1. If the conditions causing the risks are either not identified or their interactions are not
recognized.
2. If the decision points are either missed or result in an incorrect decision(s).
3. As a result of 1 and or 2, no actions are taken or incorrect actions are taken.

So far, we have shown the LOFT design process as one where the event sets, as well as initial
and changing conditions, are used to generate decision points. The decision points, if missed or
responded to incorrectly, cause a rise in the mission risk. The MPM is overlaid to give a level of
detail whereby an analysis will determine where the errors were made: in flight control, in CRM, or
in CRM and flight control. The MPM also offers an evaluative framework. However, LOFT design
could be done in the other direction.

LOFT Design: Another Approach

We have discussed selecting event sets that have built-in, as it were, decision points. Put another
way, events/event sets can be selected that require decisions (and actions) to prevent risk from.
rising—to prevent the aircraft’s position in the ops envelope from approaching a corner or a
boundary. As an example, the event set could include deteriorating Wx en route or at the destination
airport, perhaps with braking advisories or crosswinds on approach/landing. From there, the MPM
would be used to develop the flightcrew tasks and functions for PF and PNF. An initial bifurcation
could be made: one path of event sets following the correct identification of rising risk and atten-
Copyright © 2008. Taylor & Francis Group. All rights reserved.

dant risk reduction actions; the second path based on nonidentification of rising risk. Now, there
would also be subpaths, for example, showing the correct identification of rising risk but incorrect
response(s). The branching process can be repeated as needed. Thus, the ODM is the driver and the
MPM is the method used to develop functionality.
However, LOFT can be developed in a different way, still using the MPM and ODM. A series
of event sets (based on incident reports, “hangar talk,” experiences, etc.) can be selected, linked,
and the PF/PNF functions identified. These event sets we will term “expanded event sets” or “fully
articulated event sets.” By analyzing these sets, the decisional points can be identified. In fact,
“identified” is not the exact term; “selected” is more appropriate. This is because it seems clear that,
in any flight, new conditions or changes in conditions (Wx, flight system problems, etc.) will result
in changes in the aircraft’s position both in the ops envelope and on the rising risk continuum.
With the initial set of fully articulated functions and actions developed via the MPM, changes
are introduced using the ODM’s boundary conditions as guidelines. That is to say, an initial set of

72838.indb
Human Factors283in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:23:03 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 284 Human Factors in Simulation and Training

boundary conditions will be specified and used as a basis for carefully selecting changes to them
that result, if left unidentified and/or unchecked, in additive or cumulative interaction, such interac-
tions driving the aircraft toward a corner or side in the ops envelope. Of course, this means that the
risk has risen to moderate or high-moderate—even to high.
The changes to the boundary conditions should be introduced at different points in flight so that
the risk does not rise suddenly. The rationale here is that one goal of the LOFT is to keep situation
awareness high by the introduction of an ongoing series of changes, rather than a compressed set
of events that lead to an immediate abnormal ops or emergency, with limited options for the flight-
crew. If the changes in the boundary conditions are introduced over the first 1 to 1½ hours of the
LOFT session, their additive and/or cumulative interactions and impact will be sequences so that
the flightcrew’s ODM skills are tested. Thus, ODM skills are tested rather than skills at handling an
overt and immediately apparent abnormal or emergency situation, which are often trained in other
venues.
An aside: This is not to say that missed decision points as well as incorrect decisions and actions
may not lead to an abnormal or emergency situation. If that occurs, then the LOFT can also dem-
onstrate flightcrew skills in the emergency arena. However, as said, the training, to include recur-
rent or special-item training of flightcrews does provide for certain emergency training. As one
example, recently some air carriers have instituted upset recovery training (recovering the aircraft
from unusual or abnormal attitudes).
To resume: By carefully introducing boundary condition changes into the event sets, the risk
can be caused to rise from additive or cumulative interactions. As before, when we indicated how
to use the ODM to MPM LOFT design methodology, there will be a branching effect, contingent
on decisions (made, unmade, correct, incorrect) and resultant actions (taken, not taken). The LOFT
scenario must be designed to include the various pathways, so that the flight simulator can be pre-
programmed for the contingencies.
It would seem that, optimally, the OPM to MPM and the MPM to ODM methods would operate
simultaneously or in an intertwined manner. It may be fairly said that the use of the MPM and the
ODM is actually a necessary and sufficient condition of effective LOFT design
At this point in time, we believe that we have presented the ODM in sufficient detail and with
useful examples. The same can be said for the MPM. We have given references for the reader who
wants more information and exposition of either model. We have presented the framework for devel-
oping ODM/MPM-based LOFT scenario(s). The evaluation of the flightcrew in the LOFT training
session has been discussed.
It is of import to now clearly state that neither the ODM, the MPM, nor any LOFT developed
using them need have an evaluative aspect. Further, if evaluation is to be an aspect of the LOFT
session, it need not be a jeopardy situation. However, we still hold to our original view that LOFT
should have a jeopardy component.
Copyright © 2008. Taylor & Francis Group. All rights reserved.

Training
Although we have not mentioned or emphasized the training aspects of the MPM or the ODM, it
is clear that there are necessary training considerations for both. However, the MPM needs little, if
any, training in terms of flightcrew. The reason is that the CRM components are already included as
part of either initial or recurrent training. The flight-control maneuvers components are all included
in flight training/type training—and many of the flight control aspects are used in recurrency train-
ing. Additionally, these flight control tasks/subtasks are all part of the handbooks used by pilots
for each type of aircraft. Put another way, from the task through the subtask level of flight control,
pilots are familiar with and have been trained in all of it. Of more importance, the performance of
these flight control tasks, and the FAA and carrier standards to which they must be performed, are
already known to the flightcrew—they have learned and been tested in them on the ground and
been evaluated on their ability to perform the standard in the air. Therefore, for any critical task

72838.indb
Human Factors284in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:23:04 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 Civil Aviation 285

decomposition used in a LOFT, the flightcrew is well aware of the subtasks required to perform the
task. Where, then, is any training needed for understanding and use of the MPM? It would seem that
a single presentation and explanation of the MPM would suffice. There are the CRM and flight con-
trol behavior marks to consider. However, these are only of concern if the LOFT is to be evaluated
for “jeopardy.” If not, the markers and scoring scales can be distributed and explained; this process
could be incorporated into the presentation and explanation of the MPM. Two hours would suffice.
Such is not the case with the ODM. This model would require dedicated training time. Again,
there is a “however”: The boundary conditions are well known to pilots. Although it is true that the
flightcrew may never have seen the way ODM structures the boundaries, no training time is really
needed for that aspect. The rising risk continuum and the concepts of interaction among boundary
conditions/functions (with resultants that can exceed the impact of the single factors involved) can
be easily trained in a 2 to 4 hour class, with pencil-and-paper exercises. At this point in time, the
optimum use of the ODM for safety would be to automate some or all of it, and make it a call-up
part of a display. Perhaps the best concept would be to have the display come up when two or more
boundary conditions, from either the same boundary side or contiguous sides, have become active.
Such an endeavor, or any discussion of it, is far beyond the scope of this chapter.

REFERENCES
Burki-Cohen, J., Go, T. H. et al., 2003, Simulator fidelity requirements for airline pilots training and evalua-
tion continued: An update on motion requirements research. Proceedings of the Twelfth Annual Inter-
national Symposium on Aviation Psychology, Dayton, OH.
Cannon-Bowers, J. A., Salas, E., and Converse, S. A., 1995. Shared mental models in expert decision making
teams. In N. J. Castellan, Jr. (Ed.), Current Issues in Individual and Group Decision Making. Hillsdale,
NJ: Lawrence Erlbaum Associates.
Ewell, C. D. and Chidester, T., 1996. American Airlines Converts CRM in Favor of Human Factors and Safety
Training, The Flightdeck, July/August, 1996. Flight Department, American Airlines; DFW Airport).
Aviation Week and Space Technology, September 6, 1996, p. 15.
Flexman, R. H. and Stark, E. A., 1987. Training simulators. Chapter in Handbook of Human Factors. Salvendy,
Gavriel, Ed.
Go, T. H., Burki-Cohen, J., et al., 2003, The Effects of Enhanced Hexapod Motion on Airline Pilot Recurrent
Training and Evaluation, AIAA-2003-5678.
Hamman, W. R., Seamster, T. L., Lofaro, R. J. and Smith, K. M., 1993. The future of LOFT scenario design
and validation. Proceedings of the Seventh International Symposium of Aviation Psychologists. R. S.
Jensen, Ed. Columbus, OH.
Lofaro, R. J. and Smith, K. M., 2003. The finalized paradigm for operational decision-making (ODM) para-
digm: Components and placement. Proceedings of the 12th International Symposium on Aviation
Psychology. Dayton, OH.
Lofaro, R. J. and Smith, K. M., 2001. Operational decision making: integrating new concepts into the para-
digm. Proceedings of Eleventh International Symposium on Aviation Psychology. R. S. Jensen, Ed.
Copyright © 2008. Taylor & Francis Group. All rights reserved.

Columbus, OH.
Lofaro, R. J. and Smith, K. M., 2001. A paradigm for developing operational decision-making (ODM). Pro-
ceedings of 2001 SAE World Aviation Congress (WAC) Conference. Various articles in Aviation Week
and Space Technology, Vol. 15, No. 3, July 17, 2000, pp. 58–63.
Lofaro, R. J and Smith, K. M., 1999. Operational decision-making (ODM) and risk management: Rising risk,
the critical mission factors and training. Proceedings of the Tenth International Symposium of Aviation
Psychologists. R. S. Jensen, Ed. Columbus, OH.
Lofaro, R. J. and Smith, K. M., 1998. Rising risk? Rising safety? The Millennium and Air Travel special issue
of the Transportation Law Journal, Vol. 25, No. 2, University of Denver Press, Denver, CO.
Lofaro, R. J. and Smith, K. M, 1993. The role of LOFT in CRM integration. Proceedings of the Seventh Inter-
national Symposium of Aviation Psychologists. R. S. Jensen, Ed. Columbus, OH.
Lofaro, R. J., Adams, R. J., and Adams, C. A., 1992. Workshop on Aeronautical Decision-Making (ADM)
DOT/FAA/RD-92/14; Vol. I, II. National Technical Information Service: Springfield, VA.
Maurino, D., 1999. Crew resource management: A time for reflection. Chapter in Handbook of Aviation
Human Factors. Garland, Daniel, Wise, John, and Hopkin, V. David, Eds.

72838.indb
Human Factors285in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:23:04 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.
 286 Human Factors in Simulation and Training

North, D. M., Finding common ground in envelope protection systems. Aviation Week and Space Technology,
Vol. 153, No. 9, August 28, 2000.
Seamster, T. L., Hamman, W. R., Smith, K. M., Lofaro, R. J., and McDougall, W., 1993. CRM assessment
and instructor knowledge structures. Proceedings of the Seventh International Symposium of Aviation
Psychologists. R. S. Jensen, Ed. Columbus, OH.
Sprogis, H., 1997. Is the aviation industry expressing CRM failure? Proceedings of Ninth International Sym-
posium on Aviation Psychology. Rakovan and Jensen, Eds. Columbus, OH.

Federal Aviation Administration Advisory

Circulars (AC) and Regulations
AC 120-35B: Line Operation Simulations (9/6/91)
AC 120-40B: Airplane Simulator Qualification (7/29/91)
AC 120-45A: Airplane Flight Training Device Qualification (2/5/92)
AC 120-AQP: Advanced Qualification Program (8/9/91). Note: This AC has been revised and will be reissued
in early 2004.
SFAR 58: Advanced Qualification Program (5/29/03). Note: This is an extension of the original SFAR
of 1991.)
Copyright © 2008. Taylor & Francis Group. All rights reserved.

72838.indb
Human Factors286in Simulation and Training, edited by Dennis A. Vincenzi, et al., Taylor & Francis Group, 2008. ProQuest Ebook Central, 11/7/08 11:23:05 A
http://ebookcentral.proquest.com/lib/erau/detail.action?docID=381355.
Created from erau on 2021-06-22 10:32:13.