·Alisan Mackey • Susan M.
Gass
SECOND LANGUAGE
RESEARCH
Methodology and Design
Alisan Mackey
C;eorgetovvn []niversity
Susan M. Gass
Michigan State []niversity
i~ ~~o~;~~n~~~up
New York London
Copyright © 2005 by Lawrence Erlbaum Associates, lnc.
AlI rights reserved. No part of this book may be reproduced in an
form, by photostat, microform, retrieval system, or an:- other
means, without prior written permission of the publisher.
First Published by
Lawrence Erlbaum Associates, lne., Publishers
10 Industrial Avenue
Mahwah, New Jersey 07430
Reprinted 2011 by Routledge
Routledge Routledge
Taylor&Francis Group Taylor&Francis Group
711 Third Avenue 2 Park Square
New York, NY 10017 Milton Park, Abingdon
Oxon OX14 4 RN
Cover design by Kathryn Houghtaling Lacey
Library of Congress Cataloging-in-Publication Data
Mackey, Alison.
Second language research : methodology and design I Alison
Mackey, Susan M. Gass.
p. cm.
Includes bibliographical references and indexo
ISBN 0-8058-5602-1 (cloth: alk. paper) ISBN 0-8058-4249-7 (pbk. : alk.
paper)
l. Second language acquisition. 2. Second language acquisition
Research. 1. Gass, Susan M. 11. Title.
P118.2.M232005
--dc22 2004053288
CIP
Plinted in the United Sta tes of America
10 9
Contents
PREFACE xiü
1 INTRODUCTION TO RESEARCH 1
1.1. Different Types of Research 2
1.2. What isa Research Report? 5
1.2.1. Title Page 6
1.2.2. Abstract 7
1.2.3. Introduction 7
1.2.4. Methods Section 9
1.2.5. Results 13
1.2.6. Discussion / Conclusion 15
1.2.7. Notes 15
1.2.8. References 16
1.2.9. Appendixes 16
1.3. Identifying Research Questions 16
1.3.1. Feasibility 18
1.3.2. Research Questions and Hypotheses 19
1.3.3. Replication 21
1.4. Conclusion 23
Follow-Up Questions and Activities 23
2 ISSUES RELATED TO DATA GATHERING 25
2.1. Ethical Issues In Research Involving
Human Subjects 25
2.1.1. Obtaining Informed Consent
From Second Language Learners 26
2.1.2. History of Institutional Review of Human
Subjects Research, Compliance,
and Problem Solving 36
v
vi CONTENTS
2.2. Conclusion 41
Follow-Up Questions and Aetivities 41
3 COMMON DATA COLLECTION MEASURES 43
3.1. Pilot Testing 43
3.2. The Signifieanee of Data Colleetion Measures 44
3.2.1. Syntax:Japanese Passives 45
3.2.2. Interaetion Researeh 46
3.2.3. Pragmaties Researeh 47
3.3. Researehing Formal Models
of Language 48
3.3.1. Aeeeptability Judgments 48
3.3.2. Elieited Imitation 55
3.3.3. Magnitude Estimation 56
3.3.4. Truth-ValueJudgments and Other
Interpretation Tasks 58
3.3.5. Sentenee Matehing 59
3.4. Proeessing Researeh 61
3.4.1. Sentenee Interpretation 61
3.4.2. Reaetion Time 62
3.4.3. Moving Window 63
3.5. Interaetion-Based Researeh 65
3.5.1. Pieture Description Tasks 66
3.5.2. Spot the Differenee 67
3.5.3. Jigsaw Tasks 71
3.5.4. Consensus Tasks 72
3.5.5. Conseiousness-Raising Tasks 74
3.5.6. Computer-Mediated Researeh 75
3.6. Strategies and Cognitive Proeesses 75
3.6.1. Observations 76
3.6.2. Introspeetive Measures 77
3.7. Sociolinguistie I Pragma ties-Based Researeh 85
3.7.1. Naturalistie Settings 86
3.7.2. Elieited Narratives 87
3.7.3. Diseourse Completion Test (DCT) 89
3.7.4. Role Play 91
3.7.5. Video Playbaek for Interpretation 91
3.8. Questionnaires and Surveys 92
3.9. Existing Databases 97
3.9.1. CHILDES 97
3.9.2. Other Corpora 97
3.10. Conclusion 98
Follow-Up Questions and Aetivities 98
CONTENTS vii
4 RESEARCH VARIABLES, VALIDITY, 100
AND RELIABILITY
4.1. Introduction 100
4.2. Hypotheses 100
4.3. Variable Types 101
4.3.1. Independent and Dependent Variables 103
4.3.2. Moderator Variables 103
4.3.3. Intervening Variables 104
4.3.4. Control Variables 104
4.4 . Operationalization 105
4.5. Measuring Variables: Scales of Measurement 105
4.6. Validity 106
4.6.1. Content Validity 107
4.6.2. Face Validity 107
4.6.3. Construct Validity 107
4.6.4. Criterion-Related Validity 108
4.6.5. Predictive Validity 108
4.6.6. Internal Validity 109
4.6.7. External Validity 119
4.7. Reliability 128
4.7.1. Rater Reliability 128
4 .7.2. Instrument Reliability 129
4.8 . Conclusion 130
Follow-Up Questions and Activities 131
5 DESIGNING A QUANTITATIVE STUDY 137
5.l. Introd uction 13 7
5.2. Research Materials 138
5.3. Intact Classes 141
5.4. Counterbalancing 143
5.5 .Research Design Types 145
5.5.1. Correlational (Associational) Research 145
5.5.2. Experimental and Quasi-Experimental
Research 146
5.5.3. Measuring the Effect of Treatment 148
5.5.4. Repeated Measures Design 150
5.5.5. Factorial Design 151
5.5 .6. Time-Series Design 152
5.5.7. One-Shot Designs 156
5.6. Finalizing Your Project 158
5.7. Conclusion 159
Follow-Up Questions and Activities 159
CONTENTS
6 QUALITATlVE RESEARCH 162
6.1. Defining Qualitative Research 162
6.2 . Gathering Qualitative Data 167
6.2.1. Ethnographies 167
6.2.2. Case Studies 171
6.2.3. Interviews 173
6.2.4. Observations 175
6.2.5. Diaries /journals 176
6.3 . Analyzing Qualitative Data 178
6.3.1. Credibility, Transferability, Confrrmability and
Dependability 179
6.3.2. Triangulation 181
6.3 .3 . The Role of Quantification
in Qualitative Research 182
604. Conclusion 182
Follow-Up Questions and Activities 183
7 CLASSROOM RESEARCH 185
7.1. Classroom Research Contexts 185
7.2. Common Techniques for Data Collection
in Classroom Research 186
7.2.1. Observations 186
7.3. Introspective Methods in Classroom Research 201
7.3.1. Uptake Sheets 201
7.3.2. Stimulated Recall 203
7.3.3. Diary Research in Classroom Contexts 203
7.4. Practical Considerations
in Classroom Research 205
704.1. Logistical Issues to Consider When Carrying Out
Classroom Research 206
704 .2. Problematics 209
7.5. Purposes and Types of Research Conducted in
Classroom Settings 212
7.5.1. The Relationship Between Instruction
and Learning in Second Language
Classrooms 213
7.5.2. Action Research 216
7.6. Conclusion 219
Follow-Up Questions and Activities 220
CONTENTS ix
8 CODING 221
8.1. Preparing Data for Coding 221
8.1.1. Transcribing Oral Data 222
8.2. Data Coding 225
8.2.1. Coding Nominal Data 226
8.2.2. Coding Ordinal Data 227
8.2.3. Coding Interval Data 229
8.3. Coding Systems 230
8.3 .1. Common Coding Systems
and Categories 231
8.3.2. Custom-Made Coding Systems 234
8.3.3. Coding Qualitative Data 241
8.4. Interrater Reliability 242
8.4.1. Calculating Interrater Reliability 243
8.5. The Mechanics of Coding 246
8.5.1. How Much to Code? 247
8.5.2. When to Make Coding Decisions? 248
8.6. Conclusion 248
Fol1ow-Up Questions and Activities 248
9 ANALYZING QUANTITATIVE DATA 250
9.1. Introduction 250
9.2. Descriptive Statistics 250
9.2.1. Measures of Frequency 251
9.2.2. Measures of Central Tendency 254
9.2.3. Measures of Dispersion 258
9.3. Normal Distribution 261
9.4. Standard Scores 263
9.5. Probability 264
9.6. Inferential Statistics 269
9.6.1. Prerequisites 269
9.6.2. Parametric Versus Nonparametric
Statistics 271
9.6.3. Parametric Statistics 272
9.6.4. Nonparametric Tests 278
9.7. Statistical Tables 280
9.8. Strength of Association 282
9.9. Eta'and Omega' 282
9.10. EffectSize 282
x CONTENTS
9.11. Meta-Analyses 283
9.12. Correlation 284
9.12.1. Pearson Product-Moment Correlation 286
9.12.2. Spearman Rho ! Kendall Tau 290
9.12.3. Factor Analysis 290
9.13. Statistical Packages 291
9.13.1. SPSS 291
9.13.2. VARBRUL 291
9.14. Conclusion 292
Follow-Up Questions and Activities 292
10 CONCLUDING AND REPORTING RESEARCH 297
10.1. The Importance of Reporting Research 29 7
10.2. The Final Stages in Reporting Quantitative
Research 298
/0.2./. Tlie Dis'Cl/SSIOn ¿flJ'
10.2.2. Limitaaons, Future Research,
and Conclusion Sections 302
10.3. The Final Stages in Reporting Qualitative
Research 304
10.4. Reporting Combined Method (Quantitative and
Qualitative) Research 307
10.5. Checklist for Completing Reports of Research 308
10.5.1. The Research Problem and Questions 308
10.5.2. The Research Hypotheses 309
10.5.3. The Audience 309
10.5.4. The Abstract 310
10.5.5. The Literature Review 310
10.5.6. The Design of the Study 311
10.5.7. Logistics 311
10.5.8. Participants 312
10.5.9. Data Gathering 312
10.5.10. Data Analysis 312
10.5.11. Conclusions 314
10.5 .12. References 315
10.5.13. Footnotes, Endnotes, Figures,
and Tables 316
10.5.14. Author's Note ! Acknowledgments 317
10.5.15 . Postresearch Concerns 318
10.5.16. Final Touches and Formatting 318
10.6. Conclusion 320
~ollow-Up Questions and Activities 320
CONTENTS xi
APPENDIX A: SAMPLE SHORT FORM WRITTEN 322
CONSENT DOCUMENT FOR SUB]ECTS WHO
DO NOT SPEAK ENGLISH
APPENDIX B: SAMPLE CONSENT FORM FOR 323
A STUDY IN A FOREIGN LANGUAGE CONTEXT
APPENDIX C: SAMPLE CONSENT FORM FOR 324
A CLASSROOM STUDY
APPENDIX D-G: SAMPLE INSTITUTIONAL 326
REVIEW BOARD APPLICATION:
GEORGETOWN UNIVERSITY, FORMS 1-4
APPENDIX H: SAMPLE TRANSCRIPTION 342
CONVENTIONS: "]EFFERSONIAN"
TRANSCRIPTION CONVENTIONS
APPENDIX 1: SAMPLE TRANSCRIPTION 345
CONVENTIONS FOR THE L2 CLASSROOM
APPENDIX]: COMMONLY-USED FORMULJE 347
GLOSSARY 350
REFERENCES 370
AUTHOR INDEX 387
SUB]ECT INDEX 393
CHAPTER 1
Introduction to Research
What is meant by research, and how do we identify good research ques
tions? These are questions that are not always easy to answer, butwe antici
pate that by the end of this book you will be in a better position to think
about them. Trus bookis in tended to be practical in nature, aimed at those
who are involved in second language studies and second / foreign language
teaching. We recognize that many people are often put off by the word re
search, including teachers who have been teacrung for quite sorne time but
are not involved in research, and those who are just beginning in the field .
We hope to demystify the process.
The American Heritage College Dictionary defined research as "scholarly or
scientific investigation or inquiry" or as a verb "to study (something) thor
oughly" (2000). Thus, in its most basic and simplest form, research is a way
of fmding out answers to questions.
We begin by reminding the reader that we are all involved in research every
day For example, consider what is probably pan of many of our lives-being
stuck in a traffic jamo As we fmd ourselves not moving on a freeway; we ask why
trus has happened and come up with a hypothesis (e.g., because there is an acci
dent ahead, or becauseitis 5:00 P.M. on a Friday afternoon). We thenseek veri
fication of our hypothesis by waiting patiently (or impatiently) until the traillc
starts moving again. If we see an accident or the flashing lights of an emer
gency vehicle, we can confrrm or at least strengthen our hypothesis. In the ab
sence of an accident, we might conclude that it must be typical rush hour
traille. In other words, every day we ask questions, come up with hypotheses,
and seek confirmation of those hypotheses.
In this chapter, we outline what readers can expect from a typical re
search report and discuss the process of generating research questions and
formulating hypotheses. We conclude the chapter by discussing issues of
feasibility and the importance of replication in second language research.
1
2 CHAPTERl
1.1. DIFFERENT TYPES OF RESEARCH
There are many approaches to dealing with research. Two of the most com
mon are known as quantitative and qualitative, although this distinction is
somewhat simplistic as the relationship is best thought of as a continuum of
research types. Quantitative research generally starts with an experimental
design in which a hypothesis is followed by the quantification of data and
sorne sort of numerical analysis is carried out (e.g., a study comparing stu
dent test results before and after an instructional trearment). Qualitative stud
ies, on the other hand, generally are not set up as experiments; the data
cannot be easily quantified (e.g., a diary study in which a student keeps track
of her attitudes during a year-long]apanese language course), and the analy
sis is interpretive rather than statistical. As mentioned previously, this is an
overly simplistic view because one can imagine a number of variations on
this theme. In general, however, quantitative and qualitative research can be
characterized as shown in Table 1.1 (based on Reichardt & Cook, 1979).
In this book we attempt to be as inclusive as possible and cover as many
research types as possib1e.
Grotjahn (1987) pointed out that there are many parameters that can be
used to distinguish research types, including the type of data (quantitative
or qualitative), the method of analysis (interpretative or statistical), and the
TABLE 1.1
Characteristics of Quantitative and Qualitative Research
Quantitative Research Qualitative Research
• Obtrusive, involving controlled • Naturalistic and controlled
measurement observation
• Objective and removed from the • Subjective
data
• Verification oriented, • Discovery oriented
conflrmarory
• Outcome-oriented • Process oriented
• Reliable, involving "hard" and • "Soft" data
replicable data
• Generalizable • Ungeneralizable, single case studies
• Assuming a stable reality • Assuming a dynamic reality
• Close ro the data
INTRODUCTION TO RESEARCH 3
manner of data collection (experimental or nonexperimental [naturalis
tic]). He outlined six "mixed" forms, as shown in Table 1.2.
To understand the two ends of the continuum-namely "purely" quan
titative and "purely" qualitative studies-consider the following abstracts
of two research reports.
Quantitative Research
Interaction has been argued to promote noticing of L2 form in a context
crucial to learning-when there is a mismatch between the input and the
learner's interlanguage (IL) grammar (Gass & Varonis, 1994; Long, 1996;
Pica, 1994) . This paper investiga tes the extent to which learners may no
tice native speakers' reformulations of their IL grammar in the context
of dyadic interaction. Thirty-three adult ESL learners worked on oral
communieation tasks in NS-NNS pairs. During each of the five sessions
of dyadie task-based interaction, learners received recasts of their
nontargetlike question forms. Accurate immediate recal1 of recasts was
taken as evidence of noticing of recasts by learners. Results indicate that
learners notieed over 60-70% of recasts. However, accurate recall was
constrained by the level of the learner and by the length and number of
changes in the recast. The effect of these variables on notieing is dis
cussed in terms of processing biases. It is suggested that attentional re
sources and processing biases of the learner may modulate the extent to
which learners "notice the gap" between their nontargetlike utterances
and recasts. (Philp, 2003 , p. 99)
This description meets the criteria of a quantitative study: it has quanti
tative data , it analyzes the data and provides results based on statistics, and
the data were collected experimentally.
Qualitative Research
This ethnographie report "thickly describes" (Geenz, 1973) the partici
pation of ESL children in the daily classroom events of a mainstream
first-grade classroom. Data for this paper come from ayear-long study
of one classroom in an international school on a college campus in the
U.S . Using a language socialization and mieropolitical orientation, the
repon describes how, through social1y significant interactional routines ,
the children and other members of the classroom jointly constructed the
ESL children's identities, social relations, and ideologies as well as their
communicative competence in that setting. The sociocultural ecology
of the community, school, and classroom shaped the kinds of
mierointeractions that occurred and thus the nature of their language
learning over the course of the year. (Willett, 1995, p. 473)
.¡,..
TABLE 1.2
Six Mixed Forms of Research
Form of Data Method of Analysis Manner of Data CoHecHon
Experimental!
Type of Research Quantitative Qualitative Statistical Interpretative Quasi-Experimental Nonexperimental
Experimental-qualitative-interpretative ¿ ¿ ¿
Experimental-qu alita ti ve-sta tistical ¿ ¿ ¿
Experimental-quantitative-interpretative ¿ ¿ ¿
Exploratory-qualitative-statistical ¿ ¿ ¿
Exploratory-quantitative-statistical ¿ ¿ ¿
Exploratory-quantitative-interpretative ¿ ¿ ¿
INTRODUCTION TO RESEARCH 5
This abstraet uses naturalistic data (observations of students in a dass
room), provides an interpretative rather than a statistical analysis, and uses
a nonexperimental designo We address a spectrum of issues related to quali
tative research in chapter 6.
1.2. WHAT 15 A RE5EARCH REPORT?
In this section, we provide a guide for readers as to what to expect in a typi
cal artide in the second language research field, focusing primarily on
quantitatively oriented research artides. Unlike quantitative research re
ports, for which there is a relatively standard format for reporting, qualita
tive research artides are more wide ranging in terms of organization (for
more information, see ehapter 6, in whieh we diseuss qualitative research).
In this chapter our goal is to give an idea of wha t to expect in a research re
port. To that end, following is a basie skeleton of a research paper. (Chapter
10 provides detailed information for researchers concerning the writing and
reporting of their own research based on all of the areas covered in this book.)
Typical Research Paper Format
TITLE PAGE
ABSTRACT
BODY
1. Introduction
A. Statement of topic area
B. Statement of general issues
C. General goal of paper
D. Literature review
l . Historical overview
2. Major contributions to this research area
3. Statement of purpose, including identification of gaps
4 . Hypotheses
n. Method
A. Participants
l. Howmany?
2. Characteristics (male/female, proficiency level, native lan
guage, etc.)
B. Materials
