Pilot Decision-Making

54

Chapter 3: Methods
Our task as flight instructors using systematic judgemental training techniques should be to compress a lifetime of flying experience into a relatively short training program to instill good pilot judgement into the emerging private pilot

Jensen and Benel (1977)

Pilot Decision-Making Overview Much has been written about the differences between experts and novices (e.g., Benner, 1984; Ericsson & Charness, 1994; Frederico, 1995; Shanteau, 1992, etc.). There is, however,

55

little research that is primarily concerned with the decision-making processes of student pilots. In addition, there is little information regarding the best practices to teach Aeronautical Decisionmaking (ADM) to novice, general aviation pilots. The objective of this study was to determine the differences in decision-making processes in a small group of students and expert pilots by exploring the strengths and weaknesses of subjects thought processes at they attempted to resolve ADM scenarios. I used a qualitative, case study approach. Novice subjects were selected from student pilots enrolled in a Private Pilot Ground School (PPGS) course at Montgomery County (MD) Airpark. Information was also acquired from expert subjects who frequently flew a variety of aircraft from the same airport. Data were acquired during interviews with novice and expert subjects. The interviews focused on the reactions of subjects to ADM written scenarios. I selected the scenarios from an ADM instrument designed by Driskill, Weissmuller, Quebe, Hand & Hunter (1998). Each of the original scenarios was modified in an attempt to make them more relevant to student pilot subjects. Research Question The research question for this study explored the differences in pre-flight and in-flight decision-making processes between novice and expert pilots. It addressed the nature of these differences and the potential application of these findings to ADM training for student pilots. It was anticipated that the data acquired during interviews with expert and novice subjects would provide a rich source of information that could serve both to identify differences and to develop theoretical generalizations. Pilot Studies Two pilot studies, consisting of complete PPGSs, were conducted during the Spring (January through May) and the Fall (September through December) of 2000. The pilot studies were designed to establish procedural guidelines, determine the suitability of subjects, perfect ADM instructional techniques, revise scenarios and develop and clarify interview topics and

Pilot Decision-Making procedures. In addition, the pilot studies provided me with the opportunity to become familiar with coding software (Ethnograph) and to practice coding transcribed interviews. Each pilot

56

study PPGS consisted of fifteen two and a half hour classes (See Appendix B for a complete list of subject matter and ADM topics included in the PPGS). The subject matter topics selected for the PPGS were based on major topics included on the Private Pilot Written Exam.

Research Design Following conversations with my committee chair, Dr. Albert Wiswell and committee member Dr. M.G. Cline, I determined that a qualitative case study approach would be an appropriate research design for this study. I employed grounded theory coding techniques that included the coding of data (the codes for student and expert subjects can be found in Appendices F and H respectively), filing memos, analyzing the data and submitting my analysis to peer reviews during Virginia Tech Coding Seminars. These techniques, described by numerous researchers (e.g., Strauss & Corbin, 1998; Wiggins & Stevens, 1999; Yin, 1994 etc.) provided useful insights and facilitated the identification of novice and expert characteristics and themes. An examination of the characteristics and themes of novice and expert subjects was followed by six comparisons related to the pre-flight and in-flight decision-making of expert and novice subjects. These comparisons are depicted below in Figure 14.

Figure 14: Overview of Research Design

Pilot Decision-Making The Sample

57

The primary research challenge involved the student subjects and the extent to which they were able to provide data that would enable me to extract meaningful information. Choosing novice subjects during the pilot studies seemed to present a challenge since students flying experiences varied from little flying time to several students who were preparing to take their flight tests with the FAA. It was determined during the pilot studies that student subjects needed to have attained a minimum level of training and experience to be useful subjects in the study. Accordingly, students who had not soloed were not considered for inclusion in the study. In addition to the student subjects, I also selected expert subjects. Expert subjects were selected on the basis of the quantity and quality of their flight experiences. I was fortunate in locating four highly experienced professional pilots. It was anticipated that information attained from subjects would provide me with useful and pertinent data. Selection Criteria I anticipated the inclusion of a variety of student subjects that reflected variations in age, career aspirations and gender. The student subjects included in the study, however, did not reflect such differences. The ideal and actual subjects involved in the study are depicted below in Figure 15. Figure 15: Ideal and Actual Student Pilot Subjects

Factors Age Career Aspirations Gender

Ideal 2 less than 30 2 over 50 2 not aspiring to be career pilots 2 aspiring professional pilots 2 Male - 2 Female

Actual 4 between 20 and 35 4 not aspiring to be career pilots 4 Male

There were nine students enrolled in my PPGS course during the Spring of 2001. Of that number, two had recently started their flying lessons and flew rather infrequently (once every 2-3 weeks). Those students were not considered for inclusion in the study due to their lack of flight

Pilot Decision-Making

58

experiences. In addition, one juvenile (under age eighteen) was not included in the study. Of the remaining six students, five indicated they were interested in becoming involved in the study. Only four of those students, however, were able to attend every class and had attained a suitable degree of training (i.e., had soloed and were working on their cross-country training). Those individuals served as the student subjects for the study. As depicted in Figure 15, student pilot subjects did not include what the author had hoped would be an ideal distribution (one that would reflect a distinct diversity with regard to age, aviation goals and gender). All of the student pilot subjects were male and none of them had aspirations of becoming professional pilots. In addition, all were relatively close in age (ranging from early twenties to mid thirties). Expert subjects were selected based on three specific criteria. First, all of the experts had attained a minimum of 5,000 hours of total flight time. Second, the experts were all professional pilots. Finally, the experts had worked as a professional pilot in a minimum of two mediums (e.g., flight instructor, FAA examiner, airline pilot, charter pilot, etc.). All of the expert subjects met the above cited criteria. Two of the experts were designated Master Certified Flight Instructors by the National Association of Flight Instructors. A third expert had worked as a CFI and as a corporate pilot. The final expert was an airline pilot who had previously been employed as a CFI and charter pilot. Previous Training of Student Pilot Subjects ADM instruction during the PPGS was provided to students in two stages. The first stage consisted of informal ADM information provided prior to the student pilot interviews. The second stage included more formal ADM training and was provided to students following the interviews. The researcher determined that no formal ADM training would be provided to student pilot subjects prior to the ADM interviews. That was done to prevent unintentional threats to the validity of the study. It was thought that providing such training, prior to the interviews, could result in pressure on student pilot subjects to relate their ADM scenario responses (during the interviews) to specific ADM training provided during the PPGS. Prior to the interviews, student subjects received ADM training via three mediums. First, subjects received informal ADM training from their flight instructors. The extent of the ADM

Pilot Decision-Making

59

training provided to students by their CFIs varied somewhat depending primarily upon the student pilots instructor and stage of training. Second, student subjects received pre-interview ADM training through informal discussions at the beginning of each PPGS class. These discussions revolved around the experiences of student subjects during their previous week of flying. It was determined during the pilot studies (Spring and Fall, 2000) that such discussions served to build trust, contributed to student pilot buy-in to the concept of ADM and helped to build a sense of camaraderie and openness among student pilots enrolled in the PPGS. Finally, one written

scenario (Scenario 4: see Appendix E) was discussed with all student pilots enrolled in the PPGS prior to the interviews. Data All of the data were acquired from the responses of subjects to four ADM scenarios. Subjects involved in the study responded to each of the scenarios (see scenarios 1-3 and FAA video Executive Decision in Appendix E) during interviews that I conducted during the PPGS (Spring of 2001). During the first eight weeks of the PPGS, student subjects acquired primary aeronautical information. The information was acquired both from the review of subject matter in the PPGS and as part of their primary flight training. The instruction was necessary for the student subjects to appreciate adequately and comprehend fully the four ADM scenarios. The formal ADM instructional portion of the PPGS was provided to students following the interviews of the student pilot subjects. ADM Scenarios The scenarios selected for inclusion in the study were designed specifically to be pertinent to student pilots. Each of the written scenarios (scenario 1-3) included four alternatives that were provided to students following their initial reactions to the scenarios. The fourth scenario was in the form of an FAA video that depicted numerous pre-flight and in-flight decisions. Two of the written scenarios related to in-flight decision-making. One of the written scenarios related to preflight decision-making. A summary of the four scenarios presented to the subjects and the type of decisions required are provided on the following page in Figure 16.

Pilot Decision-Making Figure 16: ADM Scenarios Provided to Subjects Type of Decision Pre-Flight In-Flight
X X X X X

60

Scenario
1

Description
control tower assigns runway that would result in a substantial cross-wind during landing aircraft engine is losing RPM during flight flaps will not retract during pre-flight inspection FAA video. Executive Decision pilot makes several decisions that culminate in a major problem

2 3 4

The written scenarios were revised after the pilot studies to include additional variables that would, it was hoped, elicit supplementary information from the subjects. For example, in the second scenario an additional variable was included (potential electrical problem) that significantly affected the thought processes of several of the subjects. This second scenario is depicted in Figure 17. A complete list of all ADM scenarios can be found in Appendix E.

Figure 17: Scenario 2: Presented to Student & Expert Subjects

You are cruising at 4,500' on top of a very thin haze layer (the ground remains in sight). It has been twenty-five hours since the engine was overhauled and the pre-takeoff run-up was well within limits. The engine slowly loses RPM with no indications of oil or fuel problems. You suspect carburetor icing and pull on the carburetor heat. The engine backfires, vibrates and gradually continues to lose RPM. In addition, you notice that the ammeter indicates that the alternator may not be supplying electrical power to the battery. You are 15 miles from the nearest airport which has an operating control tower. You are in an area with moderate to heavy air traffic. You decide to:

Rank Order 1234 1234 1234 1234

Alternative a. pull out the mixture, stop the engine and check the fuel selector valve, magneto switch settings and declare an emergency. b. push in the carburetor heat, keep the engine running and divert to the closest airport c. keep the carburetor heat on and see what happens d. push in the carburetor heat, keep the engine at idle, declare an emergency and ask for advice

Pilot Decision-Making Response Format

61

The response format for the three written scenarios was developed during the second pilot study. It was determined that a greater depth and breath of information could be acquired by dividing the response format into two parts. The first part involved each subjects initial reaction to the scenario. During that stage of the interview, I attempted to acquire as much data as possible related to the subjects thought processes. In addition, I attempted to determine what additional information the subjects would like to have available, why they believed that the additional information was necessary and how they planned to use the additional information to help them make a decision. During the second part of the interview, I revealed the four alternatives to the subjects and asked them to rank order the alternatives . During that stage of the interview, I attempted to determine why the subjects rank ordered the alternatives as they did and the thought processes they used during the selection process. In addition, I attempted to determine what the subjects thought was right, wrong, silly, outrageous, etc. with each of the alternatives. Finally, I asked the subjects to develop an ideal alternative that could either be a composite of the provided alternatives, a new alternative made up entirely by the subjects or a combination of new and provided options. The response format to the FAA video scenario (Executive Decision) was also changed following the second pilot study. The video depicted the pre-flight and in-flight decisions of a private pilot. During the pilot studies, the interview was conducted after completion of the brief video. It was determined during the second pilot study, however, that the quantity and quality of the data could be enhanced by viewing the interview in two segments. The first part of the interview, therefore, was conducted following the pre-flight portion of the video. The subjects were queried about pre-flight ADM processes and how their thought processes would differ from those of the pilot in the video. The second portion of the video was then played to the subjects. The second portion of the video presented the pilots in-flight ADM. Following that portion of the video, subjects were questioned about their thoughts regarding the in-flight decisions and actions depicted in the video. The subjects frequently altered between discussing what they would do in the video scenario and what the pilot in the scenario did. This required a more detailed analysis of the data to assure that information acquired was based on the decision-making of the subjects.

Pilot Decision-Making Probing Techniques For Interviewing Novices & Experts The pilot studies served to provide me with an opportunity to refine my interviewing skills. Techniques used to probe for information were developed and interview topics were revised. In addition, new techniques and tactics were acquired during my conversations with colleagues enrolled in a coding seminar at Virginia Tech. Each interview began with an explanation of my role as an impartial researcher (as

62

opposed to a CFI). I stressed that all information obtained from each interview would be strictly confidential. In addition, each of the subjects was provided with a copy of the Virginia Tech Informed Consent document (see Appendix J). The document was reviewed thoroughly with each of the subjects prior to the first interview. Subjects were also informed that they should feel free to speak candidly and that I was primarily interested in their thought processes as opposed to their specific actions with regard to scenarios. I suggested that subjects attempted to think out loud as they considered alternatives. That process proved useful for several of the subjects. Kvale (1996) stressed the importance of developing a balance between conversation and formal question and answer sessions during qualitative research interviews. Accordingly, I attempted to establish such a balance quickly. I also attempted to provide the interviewees with a relaxed and comfortable environment that served to ease tension and to promote the collection of data. During the interviews, I periodically summarized what the subjects had said to clarify major points. I purposely attempted to avoid turning the interview into a question and answer session by asking open-ended questions and pursuing topics and issues initiated by the subjects. In addition, I purposely slowed the pace of the student subject interviews by pausing, periodically and speaking slowly. This was done to provide the student subjects with more time to consider the alternatives and to ask questions. To facilitate the flow of information, I developed a written guide. Eleven discussion topics were developed in the form of an Interview Guide that was used to assist me in the acquisition of data. The Interview Guide is depicted on the following page in Figure 18.

Pilot Decision-Making Figure 18: Interview Guide: Topics to be Discussed With Subjects During Interviews

63

Topic to be Discussed
Thought Process Thinking Priorities

Remarks
The thinking process upon which decisions are based. Subjects are encouraged to think out loud. The most important detail to consider in your decisionmaking process- 2nd most important detail to consider, etc. Is there more than one decision to be made? If so, what is the most important decision to be made? What are your priorities? How were these priorities determined? Do you view this as a stressful situation? How does stress affect/relate to your decision-making process? Was time an issue in the thought process of decision-maker? If so, how did the amount of time available to make a decision affect the thought processes of subjects What additional information would decision maker like to have available/need to know and how would this information be used in the decision-making process? Why was this additional information necessary? What is the decision maker certain of/uncertain of and how do these certainties and uncertainties affect the ADM process? What ambiguities exist and how will the case studies cope with these ambiguities? What is right, wrong, silly, etc. with each of the scenarios? What thought process was used by the subjects to rank order the alternatives? Is there an ideal alternative to each of the scenarios? What makes it better than the others? How does this ideal alternative affect the decision-making thought process? What ADM lesson(s) can be obtained from each of the scenarios with regard to the thought process of pilots? Discuss the general ADM thought processes related specifically to the scenario. How do subjects view the seriousness of the scenario? Does it present a major concern?

Decision Priority Stress

Time

Additional Information

Certainties/Uncertainties

Alternatives

Ideal Alterative Lesson

Scenario Specifics

I interviewed a total of eight subjects regarding four ADM scenarios which resulted in a total of thirty-two ADM interviews. The length of each scenario interview varied and ranged from twenty minutes to slightly over forty minutes. The average scenario interview lasted approximately thirty minutes with the total interview time for each of the eight subjects averaging

Pilot Decision-Making slightly under two hours. I provided the subjects with a fifteen minute break following the first

64

two interviews. The subjects were interviewed during a seven week period between March and May, 2001. Initial coding began shortly after the first interviews. Instrumentation Interviews were used to acquire data from novice and expert pilots. The student pilot interviews began approximately half way through the PPGS course. The reason for delaying the interviews was to assure students had acquired fundamental knowledge required to comprehend fully the scenarios. During the pilot studies the student pilot subjects were interviewed following the fifth class. I determined that several of those subjects had not acquired the basic knowledge to appreciate fully the complexities of the scenarios. I therefore attempted to assure that all relevant knowledge areas were reviewed in the PPGS prior to the interviews. To assure that was accomplished, I postponed the interview of student pilot subjects until after the eighth class of the PPGS. The specific knowledge areas related to the four scenarios and the classes in the PPGS that reviewed these knowledge areas are depicted on the following page in Figure 19. One of my major goals was to build trust with student and expert pilot subjects. I used four techniques to help build trust. First, all of the subjects were provided with a copy of an Informed Consent document (see Appendix F). I reviewed the document with subjects and stressed that their anonymity would be guarded. Second, I attempted to be non-judgemental during all interviews. A conscious effort was made to assure that subjects realized that there were no wrong responses and that they should feel free to express their true feelings. Third, I tried to demonstrate to the subjects that I was truly interested in their responses. I did this by listening carefully to what was said and by asking appropriate follow-up questions for clarification. Finally, I asked subjects to review the completed transcripts.

Field Procedures All of the subjects were tape recorded. A platform microphone was used to record all interviews. The tapes remained in my possession. Student and expert subjects were assured that their anonymity would be protected and that they would have the opportunity to review and correct transcripts. Subjects were also informed that they could elect to dismiss themselves from the study at any time.

Pilot Decision-Making

65

Figure 19: Relationship of Knowledge Areas Reviewed in PPGS to Scenarios

Scenario
1

Knowledge Areas
Student sign-off Procedures for Cross Country Flight Relevance of a Cross-wind on Take-off and Landing Basic Meteorology Airports and Traffic Patterns Communications and Air Traffic Control Aircraft Engines Communications Carburetor Icing Basic Meteorology Aerodynamics Aircraft Systems Basic Meteorology Aircraft Performance Fuel Management/Requirements Pre-flight Inspection Introduction to Flight Planning

Reviewed In PPGS Class #

1 7 8 1 3 5 3 4&5 8 2 5 8 7 7 5 1

The data were collected at Congressional Air Charters (CAC) hangar at Montgomery County Airpark. An office in the hangar was provided by the management of CAC. The office consisted of two desks and chairs and a small table. The subjects sat at one desk and I sat at the other desk. The microphone was placed between the two desks on the small table. A small TV (with videotape component) was placed on the side of the subjects desk (which facilitated viewing of the video scenario).

Data Analysis I transcribed and coded all data obtained during the interviews. The analysis of the data were facilitated through the use of grounded-theory coding techniques. These techniques assisted me in developing a more thorough understanding of the data and to attain a more complete appreciation of the thought processes of subjects.

Pilot Decision-Making

66

Open coding, a process of fracturing the data, was followed by axial and selective coding which helped me to identify characteristics and general themes. Following the initial coding, I repeated the process of axial and selective coding (combining codes and looking for characteristics and themes). Emerging characteristics and themes were included as memos in Ethnograph, a computer software program designed to assist researchers involved in qualitative studies. A review of the memos was helpful in identifying relevant distinctions between expert and student subjects. My analysis was also submitted to peer review by colleagues in several Virginia Tech coding seminar classes. Their insights and suggestions helped me to refine further developing characteristics and themes. It was anticipated that the analysis would serve to highlight differences between the two groups. Ultimately, I attempted to identify patterns of decision-making and the emergence of themes for both novice and expert pilots. Finally, I compared these patterns and attempted to develop theoretical generalizations from the data.

Limitations A significant limitation was that the student pilot subjects were all relatively low-time pilots (between 20 and 40 hours of total flying time). The results of this study, therefore, can not be generalized beyond the level of low-time student pilots. It is possible that student pilots with more experience (student pilots who have completed their training and were in the process of preparing for their FAA flight tests) may have reacted differently to the scenarios. In addition, the four student pilot subjects consisted of a relatively young group of males. There may have been a variety of responses had the study included a more diverse group of subjects. The study was further affected by geographical limitations. All of the subjects resided in the Maryland suburbs of Washington, DC. Threats to External Validity Two factors presented threats to the external validity of the study. First, I was the ground school instructor of all student subjects involved in the study. Accordingly, there was the possibility that student pilots may have been telling their instructor what they assumed he wanted to hear. I attempted to limit the effects of this threat by paying close attention to the responses of the subjects and asking detailed follow-up questions. Furthermore, I spoke to each of the

Pilot Decision-Making subjects prior to the scenario interviews. During those conversations, I emphasized to the subjects that negative comments were as important to the success of the study as were positive

67

comments and that they should feel free to speak their mind and to not be concerned about my feelings or preferences. In addition, the students selected for inclusion in the study may have become victims of the Hawthorne Effect. According to McMilan and Schumacher (1993), the Hawthorne Effect involves subjects in a study who act atypically simply because they are members of a study. The author attempted to limit the impact of the Hawthorne Effect by not referring to or acknowledging the subjects participation in the study during PPGS classes and by conducting the interviews as though they were a routine extension of the PPGS.

Summary The study examined pre-flight and in-flight decision-making thought processes of student and expert subjects. Two pilot studies provided me with valuable information regarding the interview process, selection criteria and factors to include in scenarios. A qualitative case study was determined to be the most appropriate research design and a grounded theory approach to the analysis of the data was adopted. Four experts and four student subjects were selected. Four scenarios were presented to the subjects in two parts. First, the subjects discussed their general reactions to the scenarios. Second, alternatives were presented and discussed with the subjects. The fourth (video) alternative was also presented to the subjects in two segments (pre-flight and in-flight). I transcribed and coded all of the interviews. I submitted the transcribed and coded data to peer review in my coding seminar class at Virginia Tech. Helpful insights and interpretations emerged during that process. The findings produced knowledge that would hopefully serve both to help students form a useful foundation in ADM and to promote safety. These findings are presented in Chapter Four.