CHAPTER 2

RESEARCH STRATEGIES
IN INDUSTRIAL AND
ORGANIZATIONAL PSYCHOLOGY:
NONEXPERIMENTAL,
QUASI -EXPERIMENTAL,
AND RANDOMIZED
EXPERIMENTAL RESEARCH
IN SPECIAL PURPOSE AND
NONSPECIAL PURPOSE SETTINGS
;

Eugene F. Stone-Romero

With little exception, advances in both the science referenced by a researcher and their empirical realiza-
and practice of industrial and organizational (l/O) tions (i.e., construct validity); (c) the-statistical esti-
psychology and allied disciplines (e.g., human mates derived from a study (i.e., statistical conclusion
resource management, organizational behavior, validity); and (d) the extent to which relations found
organization theory) hinge on the existence of find- in a specific study generalize across' different settings,
ings from sound empirical research (referred to units, treatments, and observations (i.e., external
hereinafter as research). The results of research are validity; Shadish, Cook, &. Campbell, 2002).
used for several purposes. One is to develop theory In view of the foregoing, the overall-purpose of
about phenomena (e.g., individual behavior in orga- this chapter is to consider the factors that influence
nizations). Another is to test predictions stemming the validity of inferences derived from research,
from such theory (e.g., expectancy theory). Yet especially those concerned with causal connections
another is to aid in the design and implementation between variables. Thus, the chapter has sections that
of interventions (e.g., job enrichment) aimed at deal with such issues as (a) study design, (b) the pur-
changing individuals, groups, and organizations poses of research, (c) the facets of validity in research,
(e.g., worker performance, group effectiveness, (d) the types of settings in which research is con-
organizational efficiency). ducted, (e) the types of experimental designs that
Whatever the purpose of research, its soundness can be used in research (i.e., nonexperimental, (
is a function of the degree to which it allows for valid quasi-experimental, and randomized experimental),
conclusions about (a). the existence of cause-effect (I) the joint consideration of experimental designs
relations between variables (i.e., internal validity); and research settings, (g) the important distinction
(b) the correspondence between the constructs between experimental design and statistical methods,
(i.e., units, treatments, observations, and settings) and (h) some conclusions about the design and

This chapter is a substantially revised version of Stone-Romero (2009), I thank the editor and the associate editors for very helpful feedback on an earlier
version of this chapter.

37
 Eugene F. Stone-Romero

conduct of research on phenomena of interest to I/O mitment) ~ Y (job performance) and (b) the rela-
psychologists. tion between X and Y differs across levels of Z (job
Note that a number of research methods-related level). In this study, the hypothesized roles of vari-
issues are covered elsewhere in this handbook. These ables are as follows: X is the independent variable,
include qualitative research (chap. 3, this volume), M is the mediator variable, Y is the dependent vari-
cross-level data analytic strategies (chap. 4, this vol- able, and Z is the moderator variable. The mediator
ume), validation strategies used in personnel selection variable transmits the effect of the independent vari-
(Vol. 2, chap. 13, this handbook), and issues associ- able to the dependent variable. And, contingent on
ated with cross-cultural research (Vol. 3, chap. 23, the level of the moderator variable, the relation
this handbook). Thus, this chapter does not offer between X and Y varies in terms of its magnitude or
coverage of these topics. form. Note that because the study is nonexperimen-
tal, the role of each of the variables is assumed. As is
explained in the next section, the researcher has no
RESEARCH DESIGN
sound basis for inferring, for example, that X causes y.
A distinction is made here between research design It may very well be the case that (a),Y causes X or
and experimental design. Research design or study (b) both X and Yare caused by an unmeasured con-
design is an overall plan for conducting a study that founding variable.
considers several components (Cook &. Campbell, Third, depending on the experimental design, the
1979; Fromkin &. Streufert, 1976; Kerlinger &. Lee, researcher must specify the strategies that will be
2000; Rosenthal &. Rosnow, 2008; Runkel &. used in manipulating or measuring variables. Fourth,
McGrath, 1972; Shadish et aI., 2002; E. F. Stone, he or she must determine how studied units, treat-
1978). First, the researcher must specify the units ments, settings, and outcomes willbe sampled
(e.g., individuals, groups, organizations) that will be (e.g., randomly versus nonrandomlyl.Fifth, he or she
studied. Second, he or she must choose from among must decide whether the study is to be conducted in
three general ways of studying relations between a special purpose or a nonspecialpurpose setting
(among) variables that differ in terms of the degree (Storie-Romero, 2002, 2009). Sixth, and finally, the
to which they provide control over possible con- researcher must specify the methods '(statistical ver-
founds. Here, the experimental design options are sus nonstatistical) that will be used in analyzing the
nonexperimental, quasi-experimental, and random- data produced by the study. As is explained in the
ized experimental designs. Note that in this chapter, next section, there is a very important distinction
experimental design refers to the degree to which a between the extent to which a study is experimental
study uses experimental methods, thus allowing and the statistical methods that are used to ~valyze
for control over confounds. Control is high with the data produced by it. Contrary to what many
randomized experimental designs, moderate with researchers appear to believe, statistical methods are
quasi-experimental designs, and very low with non- virtually always a very unacceptable substitute for
experimental designs (see the section titled Degree to randomized experiments in terms of a researcher's
Which a Design Is Experimental). ability to make valid inferences about causal connec-
In research that uses randomized experimental tions between variables (Cook &. Campbell, 1976,
or quasi-experimental designs, the investigator must 1979; Cook, Campbell, &. Peracchio, 1990; Ling, I
devise strategies for manipulating independent vari- 1982; Rogosa, 1987; Rosopa &. Stone-Romero, 2008;
ables and measuring dependent variables. And in Shadish et al., 2002; Storie-Romero, 2009; Stone-
studies of the nonexperimental variety, he or she Romero &. Rosopa, 2004, 2008). Thus, when causal
must determine how assumed independent, media- inference is important, a researcher should conduct
tor, moderator, and dependent variables will be research that uses either randomized experimental
measured. Consider, for example, a nonexperimen- designs or quasi-experimental designs that allow for
tal study in which a researcher hypothesizes that ruling out likely threats to internal validity (CookSr
(a) X (e.g.,job satisfaction) ~ M (organizational com- Campbell, 1979; Shadish et al., 2002).

38
i

i'
·il
..L.11
 Research Strategies in I/O Psychology

GENERAL PURPOSES OF RESEARCH addresses. Thus, a researcher must be concerned

with the correctness (i.e., truth value) of inferences
Research on phenomena of relevance to organiza-
associated with the four previously mentioned facets
tions and their members is conducted for a number
of validity, that is, internal, statistical conclusion,
of major purposes, most of which involve assessing
construct, and external (Cook &: Campbell, 1976,
relations between (among) unobservable constructs
1979; Cook et al., 1990; Shadish et aI., 2002). The
using manipulations or measures of variables that
better the overall design of a study, the greater the
serve to operationally define the constructs (Cook &:
confidence he or she can have about the validity of
Campbell, 1979; Guion, 2002; Nunnally, 1978;
his or her research-based inferences.
Nunnally &: Bernstein, 1994; Shadish et al., 2002;
E. F. Stone, 1978). For example, an investigator may
A Model of Research-Based Inferences
study the relation between the constructs of worker
A model of factors that influence the validity of
ability (operationally defined by scores on a measure research-based inferences is shown in Figure 2.l.
of job knowledge) and job performance (opera- Arrow 1 in the figure links the operational definitions
tionally defined by supervisor ratings). Often, he or of cause and effect constructs, and relates to internal
she may not have an immediate interest in determin- validity inferences. Arrows 2 and 3 deal with factors
ing if an observed relation is causal in nature. that affect the construct validity of measures and
However, having found a relation in one study, he manipulations, that is, relations between constructs
or she may be interested in testing for a causal rela- and their operational definitions (sampling particu-
tion in a subsequent investigation. F-;;rexample, a lars; Shadish et al., 2002). Arrow 4 is concerned with
study may deal with the effect of an intervention the correctness of statistical inferences about the rela-
designed to improve job knowledge (e.g., training) tion between the operational definitions and deals ..
on job performance. with statistical conclusion validity. Finally, Arrow 5-.
A second major purpose of research is to deter- has to do with factors that moderate the causal
mine the effects of various types of manipulations relation between X and Y, thus influencing external"
(interventions, treatments) of unobservable con- validity. Note that in the interest oflimiting the com- .
structs on criterion constructs. Some examples plexity of Figure 2.1, it considers only the constl11ct,
include the effects of (a) job enrichment on job validity of operational definitions of assumed causes
satisfaction, (b) realistic job previews on employee and effects. However, as noted in the followingsee-
turnover, and (c) preemployment interview structure tion, in an actual study the researcher also would
(structured vs. unstructured) on biases (e.g., sex,
race, age) in ratings of job applicants. In such studies,
causality is an important concern.
A third major purpose of research is to deter-
Factors that influence
mine whether causal or noncausal relations construct validity
between (among) variables that are found in a study
with a given set of units, treatments, and observa- Factors that influence
statistical conclusion validity
tions generalize across other types of units, treat-
ments, and observations. For example, is a Operational 4 Operational
definition 1 definition
stress-reduction intervention as effective for police of cause
5 of effect
construct construct
officers in a SWAT unit as it is for surgeons in a
Factors that influence
trauma center? external validity

FIGURE 2.1. Validity facets in empirical research.

FACETS OF VALIDITY IN RESEARCH From Handbook of Organizational Research Methods
(p. 305), by D. Buchanan and A. Bryman (Eds.), 2009.
A major objective in research is to generate valid London: Sage. Copyright 2009 by E. Stone-Romero,
inferences (conclusions) about the issues it Reprinted with permission.

39
Eugene F. Stone-Romero

consider the construct validity of its units, treat- Focal construct Operational definition
ments, and settings.

Construct Validity
Bias

»>
Construct validity has to do with the degree of corre-
2
spondence between the constructs referenced by a
researcher and their empirical realizations. For a
study to have a high level of construct validity, there
Unreliability
must be a high degree of correspondence between
(a) the units, treatments, observations, and settings (52
e
to which inferences are made and (b) the study's
empirical realizations (sampling particulars) of the
units, treatments, observations, and settings (Shadish
et al., 2002). Thus, for example, there would be high
construct validity of samples of an assumed cause FIGURE 2.2. The construct validity of operational
definitions of manipulations and measures. From
(e.g., worker ability) and an effect (e.g., worker per- Handbook of Organizational Research Methods (p, 304),
formance) if the manipulations and measures used by D. Buchanan and A. Bryman (Eds.), 2009. London:
in a study were valid empirical realizations of the Sage. Copyright 2009 by E. Stone-Romero. Reprinted
with permission.
underlying constructs (Cronbach & Meehl, 1955;
Guion, 2002; Nunnally, 1978; Nunnally & Bernstein,
1994; Shadish et al., 2002; Stone-Romero, 1994).
factors that detract from it in terms of variance in an
Construct validity of treatments and observations. unobservable, focal construct (aE) and its 0p'era-
Research tests for relations between variables using tional definition (a~). As shown in the figure, the
operational definitions are of two basic types, that variance shared by these is construct validity (a~).
is, manipulations (treatments) and measures Several factors detract from it. One is bias,tha~ is,
(observations). Construct validity is critical to both systematic variance in an operational definition that
such definitions. Manipulations are actions taken by is unrelated to the focal construct (a~). For-example,
a researcher to vary the value of an independent a questionnaire measure of organization~l citizen-
variable (Aronson, Ellsworth, Carlsmith, & ship behavior might be biased by participants' .
Gonzalez, 1990; Cook & Campbell, 1976, 1979; responding to its items in a SOciallydesirable man-
Cook et al., 1990; Fromkin & Streufert, 1976; ner. Construct validity also might be affected
Shadish et al., 2002). Some examples include adversely by unreliability, that is, nonsystematic
redesigning jobs, introducing quality control pro- (random) variance in an operational definition (0";).
cedures, installing a computerized human resource For instance, the just-mentioned measure might not
information system, and implementing absence be internally consistent (Nunnally, 1978; Nunnally
control policies and procedures. Measures are & Bernstein, 1994). However, even if an operational
observations of the values of assumed independent, definition is free from bias and unreliability, it may
mediating, moderating, and dependent variables. still have a low level of construct validity because
There are numerous strategies for measuring vari- it is deficient; that is, it may not fully capture the
ables including questionnaires, personality invento- essence of the focal construct (Blum & Naylor, 19))8;
ries, and aptitude and/or ability measures. These are Borman, 1991; Smith, 1976; Stone-Romero, 1994) ..
considered in the subsection titled Measuring For example, assume that a researcher studied the
Variables. relation between job enrichment and job satisfac-
Figure 2.2 is concerned with the construct valid- tion. The job characteristics model of Hackman and
ity of operational definitions of assumed causes and Oldham (1976) views enrichment as a function of
effects. It considers both construct validity and the the core job characteristics of autonomy, task signif-

40
 Research Strategies in I/O Psychology

icance, skill variety, task identity, and feedback from business school students are equivalent to man-
the job. In view of this conceptual definition, a mea- agers). In contrast, external validity deals with the
sure of job enrichment would be deficient if it only degree to which a causal relation that is found with
had items that dealt with feedback from the job. In one set of sampling particulars (business students)
Figure 2.2, deficiency is represented by the portion also is observed with other sampling particulars
of variance in the focal construct that is unshared (psychology students).
with the operational definition «)~).
The construct validity of an operational defini- Internal Validity
tion of an assumed cause or effect can be affected Internal validity has to do with thedegree of correct-
adversely by such problems as (a) an inadequate ness (i.e., truth value, legitimacy) of inferences about
preoperational definition of the construct (e.g., a con- causal connections between focal constructs (Cook
ceptual definition that underrepresents the focal con- &:. Campbell, 1976, 1979; Cook et al., 1990; Shadish
struct), (b) a Single operational definition of the et al., 2002), for example, the inference that X causes
construct (e.g., a specific measure of organizational Y (symbolically: X ---7 Y). It is important to note that
commitment), (c) a manipulation that motivates there is an important distinction between actual and
research participants to respond in a biased manner assumed causal relations between constructs (Stone-
(e.g., demand characteristics-based responding, Romero, 2009; Stene-Romero &:. Rosopa, 2004, 2008).
evaluation apprehension-based responding), and Causal relations are considered to be (a) actual when
(d) an operational definition that fails to capture they are supported by sound randomizedexperi-
the degree of variability in the focal construct mental research, and (b) assumed when based on--
(Nunnally, 1978; Nunnally &:. Bernstein, 1994; evidence from nonexperimental research, Research
Rosenthal &:. Rosnow, 2008; Shadish et al., 2002; that uses randomized experimental designs provides
Stene-Romero, 1994). Several of these problems the firmest basis for demonstrating that -an indepen-
(e.g., demand characteristics) are considered in the dent variable (X) causes (i.e., produces. changes in).
subsection titled Measuring Variables. a dependent variable (Y). The validity of causal infer-
ences is somewhat weaker in research that uses
Construct validity of units and settings. The quasi-experimental designs, and is weakest in
units that are sampled in a study have a bearing on research that uses nonexperimental designs.
construct validity. For example, construct validity In virtually all nonexperimental research, a cause
would be questionable if a study purported to deal is assumed (XA) as opposed to actual (X)~ and an
with the decision-making behavior of managers but effect is assumed (YA) as opposed to actual (Y).
used undergraduate business school students as par- However, an assumed cause can be' shown to be an
ticipants. In addition, the setting in which a study is actual cause through appropriate experimental
conducted influences construct validity. For instance, research.
a study of soldier behavior in combat that is con- Inferences about cause are most justified when a
ducted in a special purpose laboratory setting would researcher can show that (a) the cause preceded the
lack construct validity if the attributes of the study's effect in time (temporal precedence), (b) the cause
setting failed to mirror those of the target setting and effect are related to one another (covariation),
(i.e., an actual war zone). and (c) there are no rival explanations of the covari-
It merits adding that, as is explained in the ation between the cause and effect (i.e., there are no
follOwingsection, there is a nontrivial distinction confounds). These conditions are most likely to be
between the construct validity of sampling particu- fulfilled in research that uses randomized experi-
lars and external validity (Shadish et al., 2002). The mental designs and least likely to be satisfied in
construct validity of sampling particulars has to do research that uses nonexperimental designs (Cook
with the c~rrespondence between the operational &:. Campbell, 1976, 1979; Cook et al., 1990; Shadish
definitions used in the study and the constructs to et al., 2002; Storie-Romero, 2002, 2007a, 2007b,
which inferences are made (e.g., that undergraduate 2007c, 2009; Storie-Romero &:. Rosopa, 2004, 2008).

41
 .;

Eugene F. Stone-Romero

It deserves adding that there are often multiple strategies (e.g., structural equation modeling) does
causes of any given dependent variable. For example, little or nothing to make up for data that are derived
research shows that job performance is caused by from a study that is poor in terms of construct valid-
worker ability, worker motivation, and the degree to ity and internal validity. Put somewhat differently,
which the worker has a clear understanding of the data that are not worth analyzing are not worth
requirements of his or her role. Thus, a researcher analyzing well.
could conduct a randomized experimental study in
which each of these variables is manipulated inde- External Validity
pendently to determine their main and interactive External validity has to do with the degree of cor-
effects on job performance. rectness (i.e., truth value, legitimacy) of inferences
about the existence of a causal relation between two
Statistical Conclusion Validity variables across different sampling particulars of
Researchers typically use statistical methods (e.g., units, settings, treatments, and outcomes, For exam-
correlation, regression, analysis of variance) to test ple, will a goal-setting programthat has been shown
for the existence and strength of relations between to improve the performance of salespeople in retail
variables (Hays, 1994; Maruyama, 1998; Pedhazur, stores in the United States improve-the performance
1982; Snedecor &:. Cochran, 1980). For example, of comparable workers in Mexico? A frequently
using data from a nonexperimental study, they infer used analytical strategy for deriving evidence on
that there is a nonzero relation between XA and YA. external validity is meta-analysi~ (Hedges &:. Olkin,
Or using data from a randomized experimental 1985; Rosenthal, 1991). ..
study, they infer that the average score for units in a It would appear that at least one sampling particu-
treatment condition (My) is greater than the average lar should be added to the set considered by Shadish
for units in a control condition (Me). In both cases, et al. (2002), that is, time. More specifically, a causal
statistical inferences are critical to inferring that the relation may vary across time periods. For example,
studied variables are related to one another (Cook the effect of job-related ability (e.g., training based)
&:. Campbell, 1976, 1979; Cook et al., 1990; Hays, on performance may vary across time \e.g., Borman,
1994; Shadish et al., 2002). As such, the validity of 1991; Ghiselli, 1956). The effect may be relatively
the same inferences may be adversely affected by a strong during the early stages of a worker'semploy-
number of factors, including low statistical power, ment. However, it may become weaker over time
failing to meet the assumptions of statistical tests, because of changes in the roles of motivationand
and conducting large numbers of statistical tests ability as determinants of performance.
using a nominal Type I error rate that is lower than External validity is threatened by any factors that
the actual (effective) Type I error rate (Cook &:. serve to moderate (Stone-Romero &:. Liakhovitski,
Campbell, 1976, 1979; Cook et al., 1990; Hays, 2002; Zedeck, 1971) the relation between an assumed
1994; Shadish et al., 2002). Some of these factors cause and a supposed effect. Stated somewhat differ-
(e.g., statistical power) are considered in the sub- ently, external validity is an issue when there are
section titled Statistical Significance Versus the interactions between a treatment (X) and one or more
Importance of Research Results. of the other sampling particulars of a study (e.g.,
It merits adding that a large number of commonly units, settings, outcomes; Shadish et al., 2002).
used statistical methods (e.g., analysis of variance
[ANOVA], correlation, multiple regression) are spe- Types of Variables in Assumed
cial cases of the general linear model (Searle, 1971). Causal Models
In addition, although data from studies that use ran- Research aimed at testing assumed causal models (as
domized experimental designs are typically analyzed defined previously) typically considers more than a
with ANOVA, they also can be analyzed with multi- simple, two-variable sequence. For example, it may
ple regression. And, as is explained in detail in the involve assumed exogenous (i.e., independent) and
next section, the use of sophisticated data analytic endogenous (i.e., mediator, dependent) variables.

42
 Research Strategies in IIO Psychology

Behavioral
beliefs r-, Attitude toward
behavior

Evaluation
of outcomes
V Moral
beliefs
'\ Behavioral

Normative
beliefs
,.
Subjective
norms
v: intentions ~ Behavior

Motivation
to comply
V Perceived
behavioral
control

FIGURE 2.3. The theory of reasoned action. From Handbook of Organizational

Research Methods (p. 306), by D. Buchanan and A. Bryman (Eds.), 2009. London: Sage.
Copyright 2009 by E. Stone-Romero. Reprinted with permission.

Figure 2.3, based on the theory of reasoned action data, behaviors (e.g., job performance), and physi-
(Ajzen, 1988), is an illustration of such a model. It cal attributes (e.g., weight).
considers five exogenous variables (i.e., behavioral
Method for collecting data. A number of methods
beliefs, evaluation of outcomes, normative beliefs,
can be used to collect and/or record data. Among
moral beliefs, and motivation to comply), four
them are paper and pencil measures (e.g., question-
mediator variables (attitude toward behavior, sub-
naires), interviews (e.g., employment related),
jective norms, perceived behavioral control, and
observations (e.g., work behavior), mechanical or
behavioral intentions), and one dependent variable
electronic recorders (e.g., polygraphs), physical
(i.e., behavior).
sampling (e.g., blood, urine, DNA), content
analysis (e.g., coding of responses to open-ended
Measuring Variables interview questions), and searches of archives
Variables that are measured in research can be cate-', (e.g., organizational, government). It deserves
gorized in terms of such criteria as (a) type of vari- adding that Internet-based data collection methods
able measured, (b) data collection method, (c) data are being used with increasing frequency in research
source, and (d) potential for the measurement in I/O psychology and related fields (Stanton &:.
process to evoke measurement-related artifacts. Rogelberg,2002).
Type of variable measured. Measurement may Data sources. Sources of data include data sub-
focus on such variables as attitudes (e.g., job satis- jects (e.g., job incumbents, teams, organizations),
faction), values (e.g., individualism), beliefs observers of data subjects (e.g., supervisors, peers,
(e.g., stereotypes), perceptions (e.g., of job subordinates, and clients of job incumbents),
characteristics), personality (e.g., need for and organizational archives (e.g., production
~c:hievement), mental abilities (e.g., numeric), records, financial statements, Equal Employment
aptitudes (e.g., mechanical), physiological states Opportunities Commission reports). Whatever the
(e.g., arousal), physical abilities (e.g., grip variable measured in a study, it is generally wise to
strength), preferences (e.g., for benefits), contents collect data from multiple sources. Doing so can
o.f.~.rchives(e.g., annual reports of firms), demo- avert problems stemming from source-based biases.
gr;lphicinformation (e.g., age, sex), biographical For example, self-appraisals of performance tend to

43
Eugene F. Stone-Romero

overstate actual levels of performance (e.g., Harris &. Method variance has frequently been assessed
Schaubroeck,1988). with the multitrait-rnultimethod (MTMM) matrix
approach advocated by D. T. Campbell and Fiske
Potential to evoke response artifacts. Methods
(1959). However, as Kalleberg and Kluegel (1975)
for measuring variables differ in terms of the
demonstrated, the validity of inferences derived
degree to which they may evoke a number of
from MTMM analyses is contingent on the degree to
response artifacts, that is, systematic or non-
which its underlying assumptions are satisfied. For
systematic biases in responses to items in a mea-
example, MTMM analyses assume that traits and
sure. They include carelessness, SOCiallydesirable
methods are uncorrelated. However, research shows
responding, yea-saying, nay-saying, agreement
that there are often trait-method correlations. Thus,
(i.e., the tendency to agree with a statement as
they recommended that the MTMM strategy be sup-
opposed to disagreeing with its opposite, irrespec-
planted by confirmatory factor analysis (CPA). This
tive of the content of the statement), evaluation
recommendation has also been made by Lance,
apprehension, random responding, self-generated
Hoffman, Gentry, and Baranik (2008). They argued,
validity, and impression management (for more on
for instance, that in the analysis of multisource per-
response artifacts, see Nunnally &. Bernstein, 1994;
formance rating data, rater source effects may rep~e-
Webb, Campbell, Schwartz, Sechrest, &. Grove,
sent valid perspectives on ratee performance, as
1981; Weber &. Cook, 1972). In this regard, a con-
opposed to biases attributable to halo error.
siderable amount of research has been devoted to
Method biases tend to reduce the construct valid-
the reactivity of measures, that is, the tendency for ity of measures (Shadish et al., 2002) by lowering
the measurement process to affect the variables the proportion of valid systematic variance (see Fig-
being measured (e.g., see Rosenthal &. Rosnow, ure 2.2). Note, however, that some artifacts (e.g., ran-
1969; Webb et al., 1981). All else constant, the dom responding) may reduce statistical conclusion
greater the reactivity of a measure, the greater the validity by increasing the proportion of random (non-
potential for the obtained data to be biased by such systematic) variance in a measure (see Figure 2.2).
artifacts as social desirability, evaluation apprehen- Assuming that at least some of the covariation
sion, and impression management. between measures of two or more variables is attrib-
Research in I/O psychology and allied fields also utable to common methods variance, researchers
has focused on the extent to which such problems should work toward its reduction. Conway (2002)
as common methods variance, response-response and Podsakoff, MacKenzie, Lee, and Podsakoff
bias, and self-generated validity influence responses (2003) offered a number of recommendations for
.to measures (Conway, 2002; Doty &. Glick, 1998; doing this. Among them are (a) obtaining predictor
Feldman &. Lynch, 1988; Salancik &. Pfeffer, 1978). and criterion data from different sources, (b) sepa-
Method variance may result in both contaminated rating the time at which predictor and criterion vari-
measures of focal variables and spuriously high ables are measured, and (c) counterbalancing.the
relations between them and measures of other vari- order in which variables are measured.
ables (e.g., job characteristics, job attitudes). Note, It is interesting, however, that Spector (2006)
however, that there is controversy over the degree viewed common method variance as an "urban leg-
to which correlations between measured variables end." As he noted, the "urban legend that there is
are biased by method variance (cf., for example, universally shared [common method] variance in our
Salancik &. Pfeffer, 1978; E. F. Stone, 1992). In methods is both an exaggeration and oversimplifica-
addition, it is not easy to determine the degree to tion of the true state of affairs" (p. 230). Thus, instead
which observed correlations between variables are of focusing on common methods variance, he argued
attributable to (a) true covariation between them that researchers should isolate the specific hypo-
versus (b) covariation caused by method-related thesized cause(s) of spurious correlation between
bias (Nunnally, 1978; Nunnally &. Bernstein, 1994; variables and control it (them) statistically. Forcexam-
Spector, 2006). ple, the spurious correlation may be a function of

44

-----------------------------------------------~--~==-
 Research Strategies in I/O Psychology

responses that are contaminated with social desirabil- menter posits that one type of training will result in
ity. To control this threat, it could be measured and lesser learning than another. This expectancy may
its "effects" controlled statistically (e.g., by partial bias the way in which the researcher delivers training
correlation). Unfortunately, unless the variables being to trainees and, thus, the amount they actually learn.
controlled have high levels of reliability and validity, The experimenter expectancies in this situation
partialling procedures may result in upwardly biased would reduce the construct validity of the training
estimates of the relation of interest (Cook &: manipulation.
Campbell, 1979; Shadish et al., 2002; Storie-Romero, Note that demand characteristics and experi-
2007a; Stone-Romero &: Rosopa, 2008). menter expectancy effects are only two of many arti-
facts that may threaten the construct validity of a
Other Construct Validity Artifacts study'S sampling particulars. Other artifacts include
The results of research also may be biased by two using only one operational definition of a construct,
other general types of artifacts that detract from con- measuring all variables with a Single method (e.g.,
struct validity, that is, demand characteristics and questionnaires), research procedures that interrupt
experimenter expectancy effects (Rosenthal &: the routines of subjects (e.g., sleep patterns), manipu-
Rosnow, 1969; Shadish et al., 2002). Demand char- lations that lead subjects to believe that they have
acteristics are biases caused by participants' beliefs experienced treatments (e.g., job-related training)
about what is expected of them in a study (Orne, that are not as desirable as those received by others,
1962, 1969; Weber &: Cook, 1972). Consider, for and subjects in one experimental condition (e.g., a
example, a randomized, between-subjects experi- no-training control group) being exposed to treat-
mental study in a special purpose setting (e.g., a ments that were intended only for subjects in another.
university-based laboratory) in which participants condition (e.g., a training condition). Information on
are randomly assigned to conditions in which they other artifacts and strategies for dealing with them is
(a) perform one of two tasks that differ in terms of offered in such works as Aronson et al. (1990),
such characteristics as variety and autonomy and Rosenthal and Rosnow (1969, 2008), and Shadish
(b) provide ratings of these characteristics. However, et al. (2002).
prior to performing the task, a confederate of the
experimenter provides the participants with his or Timing of Measurement in Research
her (a) views about it (e.g., telling subjects that the Instudies of assumed causal models (e.g., the model
task is boring, repetitive, and mindless) and (b) affec- shown in Figure 2.3), it is critical that assumed
tive reactions to it (e.g., telling subjects that they mediators and dependent variables be measured at
won't like working on the task). In this case, their appropriate times (Mathieu, DeShon, &:. Berg, 2008;
ratings would be a function of not only the actual Mathieu &: Taylor, 2006; Shadishet al., 2002; Stone-
characteristics of the tasks but also the demand Romero &: Rosopa, 2008). The reason for this is that,
created by the information that the confederate pro- typically, the influence of causes on mediators and
vided to the subjects. As a result, the task characteris- effects is not instantaneous. For example, in a study
tics manipulation would be confounded with the aimed at testing the effects of supervisor-supplied
demand (bias) produced by the confederate-supplied feedback on worker behavior, it may take several
information. weeks for feedback (the cause) to influence the
Related to demand characteristics are experi- worker's behavioral intentions, and for these to affect
menter expectancy effects. These are biases that result the worker's behavior (i.e., job performance). Thus,
from unintentional behavior on the part of the it is critical that there be appropriate lags between
researcher (e.g., experimenter) that increases the the times at which (a) causes are varied (either natu-
likelihood of a study providing support for one or rally or experimentally) and (b) mediators and
more of its hypotheses. For instance, consider a ran- effects are measured. One way of determining the
domized, between-subjects experimental study of the appropriate lags is to base them on published reports
effects of training on learning in which the experi- of previous research on the phenomena of interest

45
 Eugene F. Stone-Romero

that provide information about time lags. Another is Sampling of Study Particulars
to conduct a pilot study to determine the time As noted previously, construct validity inferences in
needed for a manipulation to affect mediators and a study are a function of the degree to which its
dependent variables (see, e.g., Mathieu & Taylor, sampling particulars (units, treatments, outcomes,
2006; Mathieu et al., 2008). and settings) are representative of the constructs to
which inferences are made (Shadish et al., 2002).
Statistical Significance Versus the A number of sampling strategies can be used in a
Importance of Research Results study. They are illustrated here by considering
Research results are almost always evaluated in strategies for sampling individual workers (units) in
terms of the criterion of statistical significance; that an organization. Basic sampling strategies that can
is, the odds that a study's findings are a function of be used for this purpose are of two types, that is,
Type I statistical error (i.e., falsely rejecting a true probability sampling and nonprobability sampling.
null hypothesis). Statistical conclusion validity In probability sampling, workers are selected from a
hinges on the Type I error rate used in testing a population (with N members) in such a way as to
study'S hypotheses. In general, the lower the proba- ensure that the probability of selecting a sample of a
bility that a study'S results may be a function of given size (e.g., n = 5) is equal to the probability of
Type I error, the more credible are its findings. selecting any other sample of the same size. The
The capacity for a study to yield statistically sig- simplest way of doing so is to select workers ~an-
nificant results is a function of two major factors domly. This can be done, for example, by (a) assign-
(Cohen, 1988; Hays, 1994; Rosenthal & Rosnow, ing each worker in a population with N members an
2008). One is the size of the effect in the population identifying number (w; 1,2, ... n) and (b) using a
(e.g., the population correlation between two vari- computer program (e.g., Excel) to identify n values
ables, p ). The other is the size of the sample (N) of w for inclusion in the sample. It deserves adding
used in testing a statistical hypothesis. All else that probability sampling is rarely used in .research
constant, the larger the N (a) the smaller the stan- in I/O psychology and related fields.
dard error of a test statistic (e.g., the standard error More complex probability sampling strategies
------o--.f-ct-.h-e-c-o-rrelation coefficient) and (b) the greater include multistage cluster sampling and stratified ran~-----.h_ ..
the odds of rejecting the null hypothesis and dom sampling (e.g., see Cochran, 1977; Kish, 1965;
concluding that there is a statistically Significant Snedecor & Cochran, 1980). For example, in strati-
relation. fied random sampling, a researcher would specify
However, it is important to recognize that statisti- strata of employees within an organization (e.g., man-
cal Significance is not equivalent to practical impor- agerial, nonmanagerial) and then randomly select a
tance. Typically, the latter is evaluated in terms of an specific number of workers from each stratum. And .
explained variance or effect size criterion (e.g., r2, R2, in multistage cluster sampling, for instance, an inves-
cO2). For example, in personnel selection, all else con- tigator would (a) select a random sample of units in
stant, the greater the correlation between a predictor one or more clusters (e.g., private- vs. public-sector
and a criterion, the greater the proportion of variance organizations in the United States) and then (b) ran-
the predictor explains in a criterion 0. P. Campbell, domly select clusters of organizations within each of .
1976; Guion, 1976, 1991,2002). However, in some the larger clusters (e.g., health care organizations,
situations, relatively low validity coefficients may sig- educational institutions).
nal considerable practical importance. For example, Nonprobability sampling uses nonrandom strate-
as utility formulas indicate, utility (i.e., the expected gies for selecting sample members (e.g., workers)
payoff of a decision-based action, expressed in dollar from a target population. Among the many non-
terms or some other metric) can be high when a probability sampling strategies are convenience (also
validity coefficient is low, but the standard deviation referred to as accidental and haphazard) sampling,
of performance is high (Boudreau, 1991;]. P. purposive sampling of heterogeneous instances, sys~
Campbell, 1976; Cronbach & GIeser, 1965). tematic sampling, and quota sampling (e.g., see

46
 Research Strategies in I/O Psychology

Cochran, 1977; Kish, 1965; Shadish et aI., 2002; conducted empirical studies (Braithwaite, 1996;
Snedecor &: Cochran, 1980). The convenience sam- Cook &: Campbell, 1979; Shadish et al, 2002).
pling strategy involves selecting a nonrandom sam- Typically, such research is based on a model that
ple based on their availability to participate in a involves (a) observing phenomena of interest
study (e.g., samples of students in university (e.g., differences in worker performance), (b) using
classes). The purposive sampling of heterogeneous induction to develop an explanation of the phenome-
instances strategy entails selecting a nonrandom non (e.g., performance is a function of worker abil-
sample of members on the basis of the researcher's ity), (c) using deduction to make a prediction about
belief that they are diverse in terms of characteristics the phenomenon of interest (e.g., increasing ability
that might influence a causal relation between vari- will result in improved performance), (d) testing the
ables (Shadish et al, 2002). For example, a researcher prediction through empirical research (e.g., an
interested in the relation between job enrichment empirical study to assess the effects of training on
and job satisfaction in an appliance manufacturing performance), and (e) deriving conclusions about
firm might select nonrandom samples of workers in the validity of the hypothesis. The same model typi-
various jobs (e.g., machinists, electricians, janitors). cally relies on the findings of existing research on the
In systematic sampling, the researcher selects sample phenomenon of interest. As Kuhn (1970) noted, nor-
members in a methodical manner from lists of mal science involves research that is "firmly based
assumed population members. For instance, a upon one or more past scientific achievements,
researcher might select every 10th person on a list of achievements that some particular scientific commu-
the full-time employees of a firm. Finally, in quota nity acknowledges for a time as supplying the foun-
sampling, the researcher selects specific numbers of dation for its further practice" (p. 10).
sample members of different types so as to produce Note that in focusing on the method of science,
a sample that is roughly representative of a target this chapter does not provide coverage of such epis-
population. For example, in an organization, the temological perspectives as conventionalism, logical
researcher might select specific numbers of male and positivism, essentialism, and falsificationism. These
female employees in managerial versus nonmanager- and other perspectives are treated in other works
ialjobs. However, when there are too few employees (e.g., Cook &: Campbell, 1979; Rosenthal &:
of a particular type to allow for powerful statistical Rosnow, 2008; Shadish et al., 2002).
analyses, employees in specific categories can be It deserves adding that a number of criticisms
oversampled. For example, if the managerial cate- have been lodged against what have been referred to
gory included 20 women and 80 men, the researcher as established, traditional, rigorous, or normal science
might sample all women and 20 men. Having (Braithwaite, 1996; Kaplan, 1964; Kuhn, 1970)
unequal numbers of men and women in the sample research methods (e.g., Argyris, 1968, 1980; Aronson
would yield underestimates of actual relations et al, 1990; Lawler et al., 1985; Thomas &: Tymon,
between sex and, for example, monthly salary. The 1982). In general, these methods rely on the previ-
reason for this is that the point-biserial correlation ously described model of science. Among the often
coefficient is greatest when there are equal propor- overlapping critiques of rigorous scientific research
tions of cases in two groups (Ghiselli, Campbell, &: are that it (a) has little or no practical utility; (b) does
Zedeck,1981). not inspire confidence in the work of scientists;
(c) distorts important aspects of reality in organiza-
tions; (d) fails to consider phenomena that are not
EPISTEMOLOGICAL ISSUES
readily measurable; (e) has low relevance to the crite-
The focus of this chapter is on research methods that ria that are important to practitioners; (D deals with
are generally consistent with the method of science phenomena that are common knowledge; (g) is artifi-
(e.g., Braithwaite, 1996; Kaplan, 1964). This method cial; (h) is trivial; (0 is irrelevant; (j) has low opera-
assumes that general laws about phenomena can be tional validity, that is, it deals with variables that ~
established using evidence from well-designed and cannot be controlled by practitioners; (k) focuses on

47
Eugene F. Stone-Romero

"proving" the truth of hypotheses; (1) is untimely, (b) agree to the collection of data on relevant out-
that is, it deals with issues that change more rapidly comes. However, it is typically the case that managers
than the knowledge produced by scientists; and are very reluctant to allow experimental research to be
(m) involves participants who are not representative conducted in their organizations. In addition, a host of
of the populations to which generalizations are to be other factors militate against the conduct of random-
made. As a consequence, for example, Thomas and ized experiments in organizations (see, e.g., Cook &.
Tymon (1982) recommended that research be Campbell, 1976, 1979; Cook et al., 1990; Shadish
designed with the practitioner in mind and that it et al., 2002). These include, for example, faulty ran-
have high levels of five general properties or charac- domization procedures, not having a sufficient num-
teristics: (a) descriptive relevance (i.e., it deals with ber of units (e.g., individuals) to assign randomly to
phenomena that are encountered by practitioners in conditions, refusals of units to participate in experi-
organizations); (b) goal relevance (i.e., it deals with ments, treatment-related attrition from experimental
outcomes that practitioners are interested in and have conditions, and spillover of the treatment to individu-
the capacity to influence); (c) operational validity als in control conditions (Cook &. Campbell, 1979;
(i.e., it focuses on independent variables that are capa- Cook et al., 1990). For the most part, researchers
ble of being manipulated by practitioners); (d) non- have little or no influence over these factors.
obviousness (i.e., it deals with phenomena that extend It deserves adding, however, that quasi-
beyond the realm of common sense); and (e) timeli- experimental studies are sometimes possible in
ness (i.e., its results are available rapidly enough to be organizational contexts. They are especially useful
of use to practitioners who are faced with the need to in instances in which organizations conduct inter-
solve problems on a timely basis). ventions that can be studied by researchers. How-
All else constant, there is nothing whatsoever ever, as Grant and Wall (in press) noted, they are
wrong with doing research that is of value to both quite rare. More specifically, they reported that a
scientists and practitioners. However, there is no rea- search of six journals over a 25-year period revealed
son why such research cannot be conducted using that less than 1% of the studies published in the
traditional, rigorous research methods (Cook &. same journals used this design type. This is unfortu-
Campbell, 1979; Fromkin &. Streufert, 1976; E. F. nate because, as is noted in the next section, well-
Stone, 1981, 1982, 1987; Shadish et al., 2002; Weick, designed and executed quasi-experimental studies
1965). In addition, there is considerable evidence have far greater levels of internal validity than
(e.g., see Locke, 1986) showing that (a) the findings nonexperimental studies.
of experimental research conducted in special pur-
pose settings (e.g., university laboratories) general-
EXPERIMENTAL DESIGNS VERSUS
izes to nonspecial purpose settings (e.g., work
RESEARCH SETTINGS
organizations) and (b) the arrangements found in
"artificial" organizations (i.e., special purpose set- Prior to describing nonexperimental, quasi-
tings) are "strikingly similar to actual organizations" experimental, and randomized experimental designs,
(Weick, 1965, p. 204). Moreover, it deserves stress- it is important to distinguish between experimental
ing that unless research has high levels of internal, designs and research settings (Stene-Romero, 2002,
construct, and statistical conclusion validity, it is of 2007c, 2009; Storie-Romero &. Rosopa, 2008). Thus,
little importance that it has external validity. this section focuses on three setting-related issues,
Unfortunately, doing the sort of research that is that is, the "laboratory" versus "field" distinction, the
strongly advocated by various critics (e.g., Argyris, influence of setting type on internalvalidity, and the
1968,1980; Lawler et al., 1985; Thomas &. Tymon, types of realism that are important in research.
1982) requires the cooperation of top-level managers Taken together, these issues are very important in
in organizations. They must (a) allow researchers to research aimed at testing for causal relations between
design and implement interventions aimed at chang- variables. As such, they have nontrivial implications
ing outcomes that are of interest to practitioners and for both internal validity and construct validity.

48

(
 Research Strategies in I/O Psychology

The Often Inappropriate Laboratory (Cook &: Campbell, 1976, 1979; Cook et al., 1990;
Versus Field Distinction Shadish et al., 2002; E. F. Stone, 1978; Stone-
Frequently, a distinction is made between laboratory Romero, 2007 c, 2009). As such, it is generally the
and field research settings (e.g., Bouchard, 1976; case that inferences about cause are more justified
Cook &: Campbell, 1976, 1979; Cook et al., 1990; when research is conducted in SP than in NSP set-
Fromkin &: Streufert, 1976; Kerlinger &: Lee, 2000; tings (Storie-Romero, 2002, 2009; Storie-Romero &:
Locke, 1986; E. F. Stone, 1978). Regrettably, this Rosopa, 2004, 2008).
distinction is not very informative Q. P. Campbell,
1986; Storie-Romero, 2002, 2007c, 2009; Stone- Realism in Research
Romero &: Rosopa, 2008). One of the reasons for this It is important that SP settings be designed with two
is that laboratories can be set up in what are typically properties in mind. More specifically, it is essential
referred to as field settings. Thus, as Stone-Romero that they have experimental realism. In addition, it is
(2007c, 2009) noted, a more appropriate distinction often desirable for them to have mundane realism
is between special purpose (SP) and nonspecial pur- (Aronson et al., 1990; Fromkin &: Streufert, 1976;
pose (NSP) research settings. Settings of the former Weick, 1965). For example, a researcher interested in
variety are created for the specific purpose of doing studying the effects of variations in job design on crit-
research and include a laboratory room at a univer- ical psychological states and job satisfaction could
sity or a simulated work setting in an industrial park. create a special purpose setting in a university-based
In contrast, settings of the latter type are created for facility. For this study, it would be important to have
purposes other than research and include organiza- job design manipulations (e.g., autonomy, feedback)
tions created for the purpose of producing products that had desired degrees of impact on research partic-
or providing services. These include such organiza- ipants, thus ensuring the experimental realism of the
tions as Microsoft, Mercedes Benz, Matsushita, Sony, study (Aronson et al., 1990). However, it would not
and the U.S. Army. be very important for the study to have mundane
SP settings have two major attributes. First, they realism, that is, a special purpose setting that had all of
are created for the purpose of conducting research the elements common to actual work organizations.
and cease to exist when it has been completed. For example, the SP setting would not have to have
Second, they are designed to allow for the effective such features as health care benefits, retirement plans,
(unconfounded) manipulation of one or more inde- and union-management agreements. Nevertheless,
pendent variables. In general, SP settings have only the more the study had these and other elements the
a subset of the features or elements that are found in greater would be its mundane realism (Aronson et al.,
NSP settings (Aronson et al., 1990; Berkowitz &: 1990; Weick, 1965) and its construct validity. In
Donnerstein, 1982; Fromkin &: Streufert, 1976; addition, the greater would be its potential to assess
Runkel &: McGrath, 1972; Weick, 1965). the capacity of the study's assumed causal model to
explain the phenomenon of interest in a nonspecial
Setting Type and the Validity purpose (e.g., organizational) setting.
of Causal Inferences It merits adding that the just-described job design
Inferences about cause (e.g., X ~ Y) vary as a func- study also could be conducted in a nonspecial pur-
tion of the setting in which research is conducted. pose setting (e.g., an actual work organization).
Because SP settings are created explicitly for the However, for several reasons, it would typically be
purpose of conducting a study, research in such set- much more difficult to conduct it in such a setting
tings typically provides for a much greater degree of (Cook &: Campbell, 1976, 1979; Cook et al., 1990;
control over confounding (nuisance) variables than Shadish et al., 2002). One reason for this is that in
research in NSP settings. The important implication most NSP settings, it is very difficult to bring about
of this is that, to the degree that confounding vari- changes in existing organizational arrangements (e.g.,
ables can be controlled, a researcher can be more physical layout of facilities, assignment of workers
confident about the internal validity of a study to jobs, pay arid fringe benefits) that are needed to

49
 Eugene F. Stone-Romero

provide for the effective manipulation of indepen- ally may be a causal relation between X and Y, a study
dent variables (Cook &: Campbell, 1976, 1979; that used an operational definition of X that lacks
Cook et al., 1990). As a consequence, research in experimental realism may fail to provide evidence of
NSP settings that uses randomized experimental or it. In addition, to the extent that a study lacks mun-
quasi-experimental designs typically results in lower dane realism, an effect found in a special purpose set-
levels of control over extraneous or confounding ting may not be found in a nonspecial purpose setting.
variables than studies using the same types of For example, the effect of X may be present when a
designs in SP settings. Therefore, inferences about special purpose setting isolates it from all of the other
cause are typically more problematic for research in variables that are likely to be found in NSP settings.
NSP settings than for research in SP settings. However, in a nonspecial purpose setting the influ-
In general, studies conducted in NSP settings ence of X may be too weak to allow for an adequate
have higher levels of mundane realism than those test of the causal model. Cook and Campbell (1979)
conducted in SP settings. This often serves as a basis commented on this in discussing the unobtrusive
for the claim that external validity inferences are treatment implementation problem.
more appropriate for studies in NSP settings than in
SP settings. However, the legitimacy of this argument DEGREE TO WHICH A DESIGN
is suspect (Dipboye &: Flanagan, 1979; Fromkin &: IS EXPERIMENTAL . 'f

Streufert, 1976; Locke, 1986; Storie-Romero, 2002;

Experimental designs used in research are of
Weick, 1965). There are several reasons for this. One
three basic types: randomized experimental, quasi-
is that even when studies are conducted in NSP set-
experimental, and nonexperimental. These differ
tings, they typically involve nonrepresentative sam-
from one another in terms of their potential to' con-
ples of subjects, settings, and operational definitions
trol confounds and thus their value in supporting
of manipulations and/or measures. Thus, the exter-
causal inferences.
nal validity of such studies is suspect. The fact that
they were conducted in NSP settings often does Randomized Experimental Designs
nothing to strengthen external validity inferences. Studies that use randomized experimental designs
Another reason is that a major purpose of research is have three major attributes (Cook &: Campbell,
to show that there is a relation between two or more 1976,1979; Cook et al., 1990; Shadish et al., 2002;
constructs. In this regard, numerous chapters in a Storie-Romero, 2002, 2007c, 2009). Taken together,
book on generalizing from laboratory to field settings the results of studies using such designs tend to
(Locke, 1986) provide clear evidence that, typically, have high levels of internal validity.
relations that are found in SP settings are also found Manipulation of independent variables. In .
in NSP settings. The findings apply to relations research that uses randomized experimental designs,
involving numerous variables of interest to organiza- the researcher manipulates the levels of one or more
tional researchers, including attributions, goal set- independent variables. For example, in a study con-
i ting, participation in decision making, financial cerned with the effects of temperature and humidity
YI
!, incentives, reinforcement schedules, job satisfaction, on worker performance, a researcher could vary
i·l
job characteristics, and job performance. each of them experimentally. He or she would have
It deserves adding that experimental realism is to have at least two levels of each independent vari-
often crucial for studies that test assumed causal able that differed enough from one another to pro-
models. The very important reason for this is that duce changes in performance. Note that the findings
tests of such models may fail to show support for of previous research (including pilot studies) could
causal connections between variables if independent be used to determine the strength of the manipula-
variables are operationally defined in such a way as tions used in a randomized experimental study.
to have insufficient impact on study participants In studies that involve two or more independent
(Aronson et al., 1990; Fromkin &: Streufert, 1976; variables, the researcher can test for not only the
Shadish et al., 2002). Thus, even though there actu- main effects of each variable but also their interac-

50
 Research Strategies in I/O Psychology

tion (Hays, 1994). In addition, if there are enough He or she also could measure subjects' beliefs about
subjects to allow for adequate statistical power the levels of manipulations they experienced. Such
(Cohen, 1988), the researcher can have more than manipulation checks are especially important in
two levels of each variable, increasing his or her terms of inferences about the construct validity of
ability to accurately model the form (e.g., linear, the manipulations (Aronson et al., 1990; Shadish
quadratic) of the causal relations between the inde- et al., 2002).
pendent variables and the dependent variable. As is noted previously in the subsection dealing
The capacity to manipulate levels of one or more with the construct validity of treatments operational
independent variables serves to strengthen infer- definitions, measures should be both reliable and
ences about cause; that is, it leads to relatively high valid. Unless outcome measures have a sufficiently
levels of internal validity. One reason for this is that high level of reliability, statistical analyses will fail to
in a study that uses a randomized experimental provide evidence on the effects of manipulations.
design, the researcher can be confident about the This is a threat to statistical conclusion validity
cause(s) preceding the effectts) in time, that is, tem- (Shadish et al., 2002).
poral precedence. In addition, assuming a properly When tests of assumed causal models are based
conducted randomized experiment, he or she can be on data from well-designed and properly conducted
confident that changes in one or more dependent randomized experimental research, (a) causal paths
variables are the result of the manipulations, as are well-known, (b) confounds are not an issue, and
opposed to confounds. (c) model misspecification is typically not a con-
Random assignment of units to study conditions. cern. Thus, inferences about cause rest on a very
A second attribute of a randomized experimental firm foundation. This argument is predicated on the
study is that units (e.g., individuals, teams, organiza- assumption that the randomized experiments are
tions) are randomly assigned to study conditions. properly designed and executed. However, there
Assuming a sufficient number of units and effective may be instances when they "breakdown." This is
randomization procedures, the researcher can be more likely to be a problem in NSP"than SP settings
highly confident of the equivalence of the units in (Cook &: Campbell, 1976, 1979; Cook et al., 1990;
each of the conditions (in terms of mean levels of any Shadish et al., 2002). Among the many causes of this
and all measured and unmeasured potentially con- are such threats to internal validity as differential
founding variables) prior to the time that subjects are attrition across treatment conditions, history, test-
exposed to the study's manipulations. As a result, he ing, resentful demoralization, and the interaction of
or she can be confident that posttreatment levels of two or more threats. These threats are defined in the
the dependent variables for each of the conditions paragraphs that follow.
resulted from the study's manipulations, as opposed to To illustrate two of these threats, consider a
confounds (Cook &: Campbell, 1976, 1979; Cook simple randomized experiment having the following
et al., 1990; Shadish et al., 2002; Stone-Romero, 2002; design:
2007c, 2009). Note, moreover, that in a study in
R x
which the unit of analysis is the group, if a large num-
ber of groups are available for random assignment to R
experimental conditions, it is not necessary to first
Here it is assumed that subjects were randomly --
randomly assign individuals to the groups.
assigned (R) to Groups A and B and complete mea-
Measurement of dependent variables. A third sures of job satisfaction at Times 1 and 2. Those in
characteristic of studies using randomized experi- Group A were exposed to ajob enrichmentmanip-
mental designs is that levels of dependent variables ulation (X), and those in Group B served as no-
s
(outcomes) are measured. For example, in the just- treatment controls. Assume that the treatment had
described study, the researcher could measure the no effect whatsoever. However, the means for the
performance of workers in each of the conditions. measured variables had the following pattern:

51
Eugene F. Stone-Romero

OlA "" OlB < Had the researcher considered

02A"" 02B' Rosenthal & Rosnow, 2008; Shadish et al., 2002).
only the data of Group A, he or she may have falsely First, some variables are not subject to manipulation
concluded that the OlA - 02A difference was caused by (e.g., the actual age and sex of study par~icipants).
x. However, the same difference could have been Second, although it may be possible to manipulate
caused by various threats to internal validity. For some variables (e.g., the personality, physical health,
example, testing would be a credible explanation of and mental health of study participants), manipula-
the OlA - 02A difference if simply completing the mea- tions of these variables would be unacceptable on
sure at Time 1 led to a change in its value at Time 2 ethical grounds if they resulted in physical or psycho-
that was unrelated to X. And history would be a plau- logical harm to participants. Third, in research that
sible explanation of the OlA - 02A difference if fac- takes place in NSP settings (e.g., work organizations),
tors outside of the experiment (e.g., changes in a researcher may not have the ability to randomly
hourly pay rates) that were unrelated to X caused the assign units (e.g., workers, teams, plants) to treat-
difference. Differential attrition would be a plausible ment conditions. Fourth, in research in NSP settings,
rival explanation if the workers in the treatment con- it may not be possible to isolate units from one
dition who had the lowest levels of job satisfaction another (spatially), leading to the breakdown of ran- '
quit their jobs priorto the measurement of this vari- domized experiments aimed at testing assumed causal
able at Time 2. To the extent that such threats are models. Fifth, research participants may refuse to be
present in a randomized experiment, the validity of assigned to experimental conditions on a random
inferences about causal connections between vari- basis. Sixth, it may be unethical to withhold beneficial
ables would suffer. treatments from units (e.g., those in a no-treatment
To illustrate another threat to internal validity, control group). As a result of these and, other factors,
consider the following, primitive randomized researchers who are interested in examining relations
experimental design: between and/or among variables that are "assumed to
R X 02A
be causes, mediators (Stone-Romero & Rosopa, 2004,
2007), and effects may have to use either quasi-
R
experimental or nonexperimental designs (Cook &
The treatment (e.g., job design change) is intended Campbell, 1976, 1979; Cook et al., 1990; Rosenthal &
to affect 0 (e.g., job satisfaction) positively. Because Rosnow, 2008; Shadish et al., 2002).
o was not measured before X was implemented, there Computer-based simulations. Randomized
is no way of knowing if it had the intended effect. For
experiments also can be conducted using simulation-
example, assume that X actually was not effective,
based manipulations of variables. When sU,chexperi-
but statistical analyses showed that 02A > 028, This
ments are conducted on existing computer systems
might lead a naive researcher to conclude that X was
(e.g., a university-based computer), they take place
responsible for the difference. However, it could
in NSP settings. However, if a computer system were
have been an artifact of scores on 0 remaining
created specifically for the purpose of conducting a
unchanged in Group A and decreasing in Group B.
simulation, the setting would be of the SP variety.
A possible cause of this is that individuals in Group B
One type of computer-based simulation that does
could have experienced resentful demoralization as a
not involve human participants is a statistical simula-
result of not experiencing X, causing their job satis-
tion. An example of this is a study by Storie-Romero
faction to decrease.
and Rosopa (2004) that was conducted in non- a
Some limitations on the use of research using special purpose setting. They assessed the effects of
randomized experimental designs. In spite of the manipulated levels of several independent variables
value of randomized experiments for internal validity (e.g., sample size, effect size, reliability of measures,
inferences, it may be difficult or impossible to use variance of measures) on the capacity (i.e., statistical
this type of design in many circumstances (see, e.g., power) of hierarchical multiple regression to produce
Cook & Campbell, 1976, 1979; Cook et al., 1990; evidence of direct and indirect mediation. This study

52
 Research Strategies in I/O Psychology

was experimental in that the values of the indepen- delivered not only the planned treatment but also an
dent variables were manipulated in the simulation. unplanned treatment (e.g., performance-contingent
A second type of computer-based simulation is praise). In addition, statistical conclusion validity
one in which independent variables are manipulated would be reduced if supervisors did not deliver the
by a computer to assess their effects on human par- planned treatment in a systematic manner (e.g., failed
ticipants (e.g., attitudes, beliefs, emotions, behav- to give performance-contingent pay to all workers).
iors; Zickar &:. Slaughter, 2002). For example, Typically, quasi-experimental designs tend to
research in an SP setting can be used to study the have a very small number of manipulated variables.
effects of computer-based manipulations of such In addition, in quasi-experimental designs of the
variables as weather conditions (e.g., wind speed, time series variety, an independent variable may be
visibility) on simulated pilot performance. introduced and then removed several times, with
the expectation that measured levels of the depen-
Quasi-Experimental Designs dent variables will covary with these changes. How-
There are five major types of quasi-experimental ever, whatever the specific type of quasi-experimental
designs: single group designs without a control con- design, the fact that independent variables are
dition, multiple group designs that lack one or more manipulated strengthens internal validity inferences
pretest measures, multiple group designs that have a (Cook &:. Campbell, 1976, 1979; Cook et al., 1990;
control condition and use pretest measures, time Shadish et al., 2002; Storie-Romero, 2002, 2007b,
series designs, and regression discontinuity designs. 2009; Stone-Romero &:. Rosopa, 2004, 2008).
An example of each of these designs is offered in a Nonrandom assignment of units to conditions.
subsection that follows. Detailed explanations of the Unlike what is true of randomized experiments, in
five types of quasi-experimental designs are available quasi-experiments units are not randomly assigned to
in other works (e.g., Cook &:. Campbell, 1976, 1979; study conditions. For example, in a quasi-experiment
Cook et al., 1990; Rosenthal &:. Rosnow, 2008; conducted in a nonspecial purpose setting, workers in
Shadish et al., 2002). Whatever their specific nature, one operating unit of a firm may be exposed to a goal-
however, quasi-experimental designs have several setting manipulation, whereas workers in a geograph-
attributes. Taken together, they typically lead to ically remote unit may serve as no-treatment controls.
internal validity inferences that are weaker than Because intact units are used in the study, they may
those stemming from research that uses randomized differ from one another on a host of variables prior
experimental designs but stronger than those derived to the time the treatment is introduced in the first
from studies that use nonexperimental designs. unit. Such differences weaken internal validity infer-
ences (Cook &:. Campbell, 1976, 1979; Cook et al.,
Manipulation of independent variables. As is
1990; Shadish et al., 2002; Storie-Romero, 2002, ""
true of research using randomized experimental
.. 2007b, 2009).
designs, in research using quasi-experimental
designs, the researcher manipulates one or more Measurement of assumed dependent variables.
independent variables. Note that in organizational In quasi-experimental research, the researcher mea-
research it is often true that the manipulations are sures the values of one or more assumed dependent
designed and introduced by both the researcher and variables. The assumed qualifier is used because
representatives of the organization. Regrettably, this units are not randomly assigned to study conditions.
may threaten both the construct validity of the ~s a result, in research using most types of quasi-
manipulations and the statistical conclusion validity experimental designs, there is no assurance that
of a study'S findings. For example, construct validity a~ross-condition differences in the measures of
would be threatened if the design of a study called for assumed dependent variables were the actual effects
all individuals in a treatment condition to receive a of the treatments. For example, they may have been
".specified treatment (e.g., performance-contingent a product of one or more unmeasured confounding
pay), but supervisors responsible for the delivery of it variables (e.g., history).

53
Eugene F. Stone-Romero

Examples of quasi-experimental designs. One threats to internal validity. For example, history can
type of quasi-experiment is a single group design be controlled by comparing pretest and posttest
without a control condition. An example of this is measures in Groups A and B. If, for e.{Cample,OIB =
the one-group pretest-posttest design: 02B = 03B, such threats as testing, history, and matu-

ration can be ruled out. This strengthens internal

x validity inferences.
A fourth type of quasi-experimental design is the
Because of its Simplicity, this design may seem time series design. An example of this is the simple
attractive to a researcher. However, in most cases, interrupted time series design:
the finding of a hypothesis consistent difference
between OIA and 02A would not serve as a valid basis OIA 02A 03A ... 02SA X
for arguing that it was caused by X. The principal 026A 027 A 028A ... 0SOA.

reason for this is that the design does not allow In this design, members of a Single group are
the researcher to rule out most of the previously measured at multiple pretest (e.g., 25) periods,
described threats to internal validity. As such, from exposed to a treatment (X), and measured at multi-
an internal validity perspective this design has virtu- ple posttest (e.g., 25) periods. These measures allow
ally no value. the researcher to rule out such threats to interval
A second type of quasi-experiment is the multiple validity as testing, maturation, and instrumentation.
group design that lacks one or more pretest measures. As a result, causal inferences are far mo~e justified
An example of this type of design is the posttest-only with this design than the three previously' described
design with a nonequivalent control group: quasi-experimental designs.
The fifth type of quasi-experimental design is the
-R x
regression discontinuity design. It has the ~ollowing
structure:

Here, units are assigned to conditions on a non-

random (-R) basis. This design allows the researcher - R OIB 02B.
to compare the measured values of the assumed out-
Units (e.g., individuals) are measured at Time 1, and
come for Groups A and B. However, even if they are
their scores are used to assign them to either the
hypothesis consistent, the researcher has almost no
treatment (A) or control (B) condition. For example,
ability to argue that the difference was caused by x.
units who scored above the mean on 01 are given
One reason for this is that there is no legitimate
the treatment, whereas the remainder serve as no-
basis for assuming that the groups were equivalent treatment controls. Subsequent to treatment, the
to one another prior to the time that the treatment researcher regresses O2 scores on 01 scores separately
(X) was delivered to members of Group A.
for units in Groups A and B. A treatment effect, for
A third type of quasi-experimental design is a example, would be Signaled by a difference in the
multiple group design that has a control condition regression coefficients for the groups. Although a
and uses one or more pretest measures. An illustra- researcher might infer that X was responsible for this
tion of this is the untreated control group design difference, the same inference may not be valid. One
with dependent pretest and posttest samples and a reason for this is that there may have been differ en-
double pretest: tiallevels of mortality (attrition) in Groups A and B.

-R OIA OlA
Some limitations on the use-ef-quasi-experimental
- R OlB 02B
designs. A number of conditions limit the capacity
of quasi-experimental research to provide sound
Note that although units were assigned to Groups A tests of causal models. They include most of the fac-
and B on a nonrandom basis, the existence of multi- tors noted previously that pertain to randomized
ple pretests allows for an assessment of several experimental designs. In the interest of brevity, the

54
 iJ
Research Strategies in I/O Psychology

same factors are not considered in this subsection. including case studies, ethnographies, and grounded
Note, however, that it is often possible to do quasi- theory (Bott, 2007; Locke &: Golden-Biddle, 2002;
experimental research that has reasonably high levels Shadish et al., 2002; Yin, 1989). In the case study, for
of internal validity, especially when a researcher can example, the researcher conducts an in-depth, typi-
demonstrate that the pattern of findings stemming cally subjective, examination of one or more units
from a study are consistent with an a priori set of (individual, group, organization) for the purpose of
expectations (see the discussion of the coherent pat- describing phenomena of interest (e.g., communica-
tern matching strategy in Shadish et al., 2002). In tion processes, maturation of units). In general, the
addition, as is detailed by Grant and Wall (in press), data produced by such research are not quantitative,
quasi-experimental studies have a number of benefits rendering them unsuitable for statistical analyses.
that should motivate their use when randomized
Measurement of assumed independent variables.
experiments are not possible, including minimizing
In nonexperimental research, variables that are
ethical dilemmas and facilitating cooperative
assumed to be causes are measured, as opposed to
research with practitioners.
manipulated. For example, a researcher interested in
studying the relation between job enrichment and job
Nonexperimental Designs
satisfaction could measure the former variable, oper-
Of the three general types of designs considered in
ating on the assumption that differences in observed
this chapter, nonexperimental designs are the most
levels of job enrichment reflect actual differences on
frequently used in organizational research (Austin,
this variable. However, because job enrichment is
Scherbaum, &: Mahlman, 2002; Scandura &:
measured, as opposed to manipulated, inferences
Williams, 2000; Stene-Romero, Weaver, &: Glenar,
about a causal connection between it and job satisfac-
1995). Research that uses such designs has several
tion would rest on a very shaky empirical foundation.
defining attributes. Taken together, they greatly
Stated somewhat differently, the same study would
reduce the validity of causal inferences (Cook &:
not have a high level of internal validity. In addition,
Campbell, 1976, 1979; Cook et al., 1990; Shadish
as is detailed in a subsection that follows, inferences
et al., 2002; Storie-Romero, 2002, 2007a, 2009;
about cause would not be strengthened whatsoever
Stone-Romero &: Rosopa, 2004, 2008).
by the application of so-called causal modeling proce-
There are two major types of nonexperimental
dures to data derived from the study. This point is
designs: quantitative and qualitative. In research of
echoed by methodologists in numerous academic
the quantitative variety, various variables (e.g.,
disciplines (Brannick, 1995; Cliff, 1987; Freedman,
assumed independent, moderating, dependent) are
1987; Holland, 1986; Ling, 1982; Mathieu et al.,
measured and quantitative estimates of population
2008; Millsap, 2002; Rogosa, 1987; Rosopa &: Stone-
parameters (e.g., mean, variance, covariance, cor-
relation) are estimated for individuals in one 0)." Romero, 2008; Stone-Romero, 2009; Stone-Romero
&: Rosopa, 2004, 2007, 2008).
more groups. Nonexperimental studies that consider
relations between (among) variables are often erro- Nonrandom assignment of units to assumed
neously referred to as "correlational studies" (e.g., conditions. In nonexperimental studies, units
Aronson et al., 1990; Shadish et al., 2002). As indi- (e.g., workers) are not randomly assigned to study
cated by Stone-Romero (2002), however, the same conditions. Rather, the researcher collects data from
label is inappropriate because correlation is a statis- units that have levels of assumed independent vari-
tical technique that is used to estimate the strength ables that may have resulted from various unknown
and direction of relations between variables. It is not causes. For example, in the just-described job enrich-
an experimental design type. ment study, the researcher would have to assume that
A second type of nonexperimental design is quali- the self-reported levels of measured job enrichment
tative research. This type of design is considered in corresponded to the actual levels of enrichment of
depth in chapter 3 of this volume. Note, however, workers' jobs. This assumption would, for example,
that there are several subtypes of qualitative studies, be erroneous if the self-reports varied systematically .

55
Eugene F. Stone-Romero

as a function of factors other than objective levels of As a result, the internal validity of longitudinal studies
enrichment (e.g., the cognitive ability levels of sub- of the nonexperimental variety is typically quite low.
jects; see E. F. Stone, Stone, &. Gueutal, 1990). Thus, frequently expressed cans for longitudinal
research as a means of testing causal propositions are
Measurement of assumed dependent variables.
ill-advised. Nevertheless, some researchers (e.g.,
As is true of research using quasi-experimental
Frese, Garst, &. Fay, 2007) believe that causal infer-
designs, studies using nonexperimental designs deal
ences are aided by the analysis of longitudinal data
with measured levels of what are assumed to be
using time-lagged structural equation modeling
dependent (outcome) variables. As a result, studies
(SEM). Regrettably, as is noted in the following para-
using nonexperimental designs have very low levels
graphs, unless causes are manipulated, the internal
of internal validity.
validity of such research is suspect; that is, the use of
Some limitations on the use of nonexperimental so-called causal modeling methods does virtually
designs. In general, research that uses non- nothing to enhance the internal validity of findings
experimental designs is easier to conduct than stemming from nonexperimental studies. Moreover,
research that uses either randomized experimental as Frese et aI. (2007) noted, their "longitudinal study
or quasi-experimental designs (Cook &: Campbell, cannot rule out the existence of unknown and
1976, 1979; Cook et aI., 1990; Shadish et aI., 2002). changing third [confounding] variables" (p. 1099).
This is especially true of studies conducted in NSP Note, moreover, that if an unknown variable
settings (e.g., work organizations). One important (e.g., X3) is responsible for covariation between two
reason for this is that organizations (and other types other variables (e.g., X, and X2) that are measured
of social systems) are far more willing to allow for longitudinally (at Times 1,2,3, ... g), the spurious
studies in which assumed causes are measured as correlation between Xl and X2 (i.e.,'fIZ) will be
opposed to manipulated. However, because of their observed at all such time periods. For example, if
questionable internal validity, nonexperimental exposure to stressful environmental conditions in a
designs should be avoided by researchers who are war zone is responsible for a spurious observed cor-
interested in testing causal models. relation between depression (Xl) and anxiety (X2),
the spurious rl2 correlation will be seen at all time
Value of longitudinal, nonexperimental
periods in which Xl and X2 are measured. Collecting
designs. In longitudinal, nonexperimental
data on Xl and X2 at three or more periods will do
research, assumed causes, mediators, and effects are
nothing whatsoever to reduce the spuriousness. In
measured. Because of this, it is Virtually always the
addition, the use of sophisticated data analytic tech-
case that two of the three requirements for making
niques (e.g., SEM) will be of no value whatsoecer in
causal inferences (i.e., temporal precedence and rul-
terms of ruling out the operation of one or more
ing out rival explanations) cannot be satisfied (Cook
confounding variables. In short, longitudinal
&: Campbell, 1976, 1979; Cook et aI., 1990; Shadish
research of the nonexperimental variety does
et al., 2002; Rosopa &. Storie-Romero, 2008; Stone-
nothing to aid in valid causal inference. However,
Romero, 2002, 2007c, 2009; Stene-Romero &.
because of design considerations (i.e., the manipula-
Rosopa, 2004, 2007, 2008). For example, McDonald
tion of independent variables), longitudinal data are
(1999) noted that longitudinal studies are
of considerable value in making causal inferences in
open to the objection that an apparently well-designed quasi-experimental studies.
causal relation between an earlier and a
later measure may just represent a rela-
INTERSECTION OF RESEARCH SETffNG .
tion of temporally stable traits of the
AND EXPERIMENTAL DESIGN
person, or perhaps aspects of an unfold-
ing developmental sequence that does As noted previously, in any given study, a researcher
not allow conceptual manipulation of has to make decisions about a number of important
the earlier measured attribute. (p. 370) issues, including its experimental design and setting.

56
 Research Strategies in I/O Psychology

. ,
This section considers the combinations of these Internal Revenue Service, law enforcement agen- ,.
rwo design features. The strengths and weaknesses of cies). They then responded to structured interyiew.
such combinations are considered by Stone-Romero items dealing with the dependent variables in terms
(2009). of a specific type of organization. Results of univari-
ate and multivariate analyses revealed that organiza-
Randomized Experiments in NSP Settings tion type affected reports of privacy-related values,
It is possible to conduct randomized experiments in beliefs, and attitudes.
NSP.However, for reasons considered previously,
studies of this type are relatively rare. Nevertheless, Randomized Experiments in SP Settings
several factors may serve to enhance the likelihood of Randomized experiments are frequently conducted
doing experimental research in NSP settings (Cook in SP settings. One important reason for this is that
and Campbell, 1976, 1979; Cook et al., 1990; Shadish they allow the researcher to manipulate independent
et al., 2002). The odds of doing it increase when variables while controlling for possible confounds
(a) the demand for a given treatment (e.g., a labor- through strict control over the study's setting.
saving device) is greater than its supply; (b) an inno- Although the label is poor (Stone-Romero, 2007c;
vation (e.g., new computers) is to be introduced, but Stone-Romero &. Rosopa, 2008), the term laboratory
it cannot be delivered to all units at once; (c) units are experiment is commonly used to describe research
isolated from one another in time (e.g., in basic train- that uses this type of design (e.g., Fromkin &.
ing units in the military); (d) units are separated geo- Streufert, 1976; Locke, 1986).
graphically and the level of inter-unit communication A study by D. L. Stone and Stone (1985) provides
is low (e.g., fast food restaurants in various regions of an example of a randomized experiment in a special
the country); (e) there is a need for change and multi- purpose setting. They were interested in factors that
ple treatments of unknown efficacy can be introduced; affected individuals' beliefs about the accuracy of per-
(f) units can be assigned to conditions randomly as formance feedback and their self-perceived task com-
opposed to on the basis of need or merit; (g) units petence (i.e., task-based esteem). On the basis of
have no preferences for the type of treatment they are relevant theory and research, they hypothesized that
to receive; (h) it is possible to create an organization self-esteem would be influenced by both the favor-
for the sole purpose of conducting an experimental ability of feedback and its consistency. To test these
study; (0 the organization gives the researcher control hypotheses, they conducted a randomized experimen-
over units for the purpose of a study; and (j) units tal study in a special purpose setting (i.e., a university
expect to be assigned treatments on a random basis. laboratory facility). The study involved role-play-
Note that in some of the just-noted cases, what was based manipulations of (a) feedback favorability
originally a nonspecial purpose setting is temporarily (acceptable vs. superior) and (b) the consistency of
converted to an SP setting for the duration of a study. feedback (consistent vs. inconsistent) from two feed-
Research by E. F. Stone, Gueutal, Gardner, and back agents. After working on an in-basket task, sub-
McClure (1983) illustrates the use of a randomized jects were randomly assigned to one of the four study
experimental design in a nonspecial purpose setting. conditions. They then received performance feedback
They were interested in the degree to which the tyfte and completed measures of the dependent variables.
of organizations with which data subjects had dealings Results of analyses of variance showed support for
(the independent variable) influenced their privacy- virtually all of the hypothesized relations, including
related values, beliefs, attitudes, and several other those concerned with interaction effects.
Outcome variables (the dependent variables). Study
participants were randomly selected from several Quasi-Experiments in SP Settings
geographical regions within the state of Indiana and Although it is possible to conduct quasi-experimental
randomly assigned to conditions in which they were research in SP settings, studies of this type appear
asked to consider their dealings with one of six nonexistent in I/O psychology and allied fields.
types of organizations (e.g., their employer, the A search of the social science literature using the

57
Eugene F. Stone-Romero

PsycINFO database from 1872 to June 2007 failed psychology and related fields. Research of this type
to reveal a single published study of this nature. is far more common than randomized experiments
However, quasi-experimental research in SP settings in such settings. One important reason for this is
is notat all uncommon in other disciplines. For that quasi-experiments are far less disruptive of
example, it is found in clinical psychology ongoing organizational structures, processes, and
(e.g., Blanchard et al., 1997). practices than randomized experiments.
Research by Blanchard et al. (1997) provided an A study by Hackman, Pearce, and Wolfe (1978)
example of a study that used a quasi-experimental illustrated the use of a quasi-experimental design in
design in an SP setting. They were interested in an NSP setting. They were interested in the effects of
determining the effects of thermal biofeedback inter- changes in job design (the independent 'Variable) on
ventions for the treatment of vascular headaches. measures of several job characteristics, job perfor-
The research took place in an SP setting, that is, a mance, and absenteeism (the assumed dependent
laboratory created for the study of stress and anxiety variables). To determine the effects of such changes,
disorders and associated treatments. There were they conducted a quasi-experimental study using
four experimental conditions: (a) thermal biofeed- 94 employees of a bank. Data on the assumed
back for hand warming (TBFHW), (b) thermal dependent variables were collected before and after
biofeedback for hand cooling (TBFHC), thermal changes in the job characteristics. Results showed
biofeedback for temperature stability (TBFTS), and that the job design manipulations resulted in
(c) suppression of alpha brain waves (SABW). To expected changes in measures of several job charac-
assess the effects of these treatments, the researchers teristics and absenteeism. However, changes in per-
conducted a randomized experiment in which formance between the pre- and postintervention
70 patients were randomly assigned to one of four periods only were found for employees who were
conditions. Results of repeated measures t tests relatively high in terms of growth need strength.
showed reductions in headache intensity for all but The researchers noted that the results of their study
the subjects in the TBFTS condition. needed to be replicated using studies with random-
Although the overall experimental design was ized experimental designs.
randomized experimental, within each of the treat- ;,t--.
D'
'./

ment conditions the design was quasi-experimental Nonexperiments in NSP Settings

because subjects who completed premeasures of , Nonexperimental studies in NSP settings are very
headache intensity were exposed to the treatment common in organizational research. In fact, Stone-
and then completed postmeasures of headache Romero et al. (1995) showed that for 1,929 articles
intensity. In addition, Blanchard et al. conducted published in the Journal of Applied Psychology for the
within-condition internal analyses (Aronson et al., period of 1975 to 1993, the percentage of studies
1990) with subsets of the subjects in the thermal that used nonexperimental designs was much
biofeedback conditions who they referred to as greater than that of studies using either randomized
learners (i.e., those who demonstrated a criterion experimental or quasi-experimental designs. Similar
level of proficiency at controlling their hand temper- findings have been reported by both Austin et al.
ature). The use of the internal analyses converted (2002) and Scandura and Williams (2000).
what was initially a randomized experiment to a Research by Storie-Romero, Stone, and Hyatt
quasi-experiment. Results of the same analyses (2003) provided an example of the use of a non-
showed that subjects in the TBFHC and TBFTS con- experimental design in an NSP setting. They were
ditions showed reductions in headache intensity, interested in determining the degree to which indi-
whereas those in the TBFHW condition did not. viduals viewed 12 personnel selection procedures
(e.g., application blank, interview, mental ability
Quasi-Experiments in NSP Settings test, work sample) to be invasive of theirprivacy. To
There are numerous instances of studies using obtain data on this issue, they conducted two non-
quasi-experimental designs in NSP settings in I/O experimental studies in-NSP settings. In Study 1, .:

"
58
 Research Strategies in I/O Psychology

Thurstone-type scaling procedures were used to pendence, (b) performed a task in a special purpose
obtain invasiveness scale scores for each of the setting (i.e., a university-based laboratory room), and
12 procedures, and in Study 2, participants provided provided perceptions of the characteristics of the task.
direct ranks of their invasiveness. In addition, in Note that all subjects performed the same task. Thus,
Study 2, participants ranked the 12 procedures in Stone reasoned that variations in perceptions of task
terms of a number of factors that were viewed as characteristics would be a function of differences in
possible antecedents of invasiveness (e.g., procedure field independence. Among the various findings were
reveals negative information, procedure invades the that field independence correlated positively with per-
body, procedure erroneously discredits applicants). ceptions of task variety, task identity, and feedback in
Results of Study 1 revealed considerable differences both studies. In addition, several other correlation
in the relative invasiveness of the procedures. In coefficients were of modest magnitude (e.g., r = -.24)
addition, the results of Study 2 were highly consis- but were not statistically significant.
tent with those of Study 1. Moreover, Study 2
showed a number of very strong correlations
STATISTICAL METHODS VERSUS
between invasiveness and its assumed antecedents.
EXPERIMENTAL DESIGN

Nonexperiments in SP Settings The joint consideration of experimental design and

Nonexperimental research is often conducted in SP statistical methods gives rise to an extremely impor- .
settings. A common example of this is questionnaire- tant point: The type of experimental design used in a
based research conducted in SP facilities that are study is independent of the statistical methods that
dedicated to research (e.g., a university-based can be used to analyze data from its use.Thus, for
behavioral laboratory). example, data from a simple randomized experiment
Independent of the experimental design used in a involving a control group and a treatment group can
study and the type of setting (SP vs. NSP) in which it be analyzed with a number of statistical methods,
takes place, a researcher must determine the degree including (a) an independent groups t test; (b) analy-
to which its data provide support for one or more sis of variance; (c) bivariate correlation, using
hypotheses. In quantitative (as opposed to qualita- dummy codes for the independent variable; (d) ordi-
tive) research, this determination is typically based nary least-squares (OLS) regression; (e) path analysis
on the results of statistical tests. Thus, the next sec- (PA); and (I ) SEM. Note, in addition, that the F
tion considers the distinction between statistical stemming from the ANOVA will equal the squared t
methods and experimental design. Of particular from a test of mean group differences. In addition,
interest and importance is the use of the findings of the estimate of the proportion of variance explained
statistical tests as a basis for supporting inferences by the ANOVA and the t test will equal the r2 derived
about causal relations between (among) variables. from the bivariate correlation analysis. What's more,
A study by Stone (1979) illustrates the use of a the F from the ANOVA will equal the F for the test of
nonexperimental design in a special purpose setting. the significance of R2 derived from an OLS analysis.
A considerable body of research shows that percep- Moreover, in PA using a correlation matrix as the
tions of various types of stimuli are related to field input, the path coefficient from the independent
independence, that is, an individual difference dimen- variable to the dependent variable will equal the
sion that deals with the ability of individuals to per- bivariate correlation coefficient and will result in the
ceive stimuli independent of the context in which they same t value as that obtained from the t test.
are embedded. On the basis of a literature review, Note that statistical methods and experimental
Stone hypothesized that field independence (i.e., an design are separate issues in terms of inferences
assumed cause) would be related to individuals' per- derived from a study. Consider, for example, a sta-
ceptions of task characteristics (i.e., assumed effects). tistical test of the difference between the mean level
Thus, he performed two nonexperimental studies in of a variable for two groups. It only provides infer- .
which subjects (a) completed a measure of field inde- mation on the degree to which a test statistic (e.g., t)
Eugene F. Stone-Romero

differs from a value that would be expected by and Human Decision Processes) in the last 4 decades.
chance, given the sampling distribution associated Indeed, several studies have shown that the use of
with the test (e.g., difference between independent SEM for testing structural and/or measurement
means, assuming equal population variances). The models has increased markedly since software for
findings of the same statistical test (t) say nothing performing such tests has become available (Aguinis,
whatsoever about the design that was used in the Pierce, Bosco, &:. Muslin, 2009; Austin et al., 2002;
study of two groups. It may have been a nonexperi- Stone-Romero et al., 1995). In view of this, it is
ment, a quasi-experiment, or a randomized experi- important to consider the conditions that are vital
ment. Thus, inferences about statistical significance to valid inferences about causal relations between
would be independent of inferences about causes of variables.
the difference between the means (i.e., the internal
validity of the study). Conditions Vital to Inferences
One very important implication of the foregoing About Cause
is that a test of an assumed causal model that uses All else constant, internal validity isa function of
data from a nonexperimental study and statistical the experimental design of a study. The reason for
methods that purport to yield information about this is that the conditions vital to causal inference
causality does nothing whatsoever to change the (i.e., temporal precedence, correlation, and absence
nature of the study's experimental design. Stated of confounds) are most likely to be satisfied in
somewhat differently, causal modeling methods are research that uses randomized experirri;ental designs,
incapable of converting data from a nonexperimen- less likely to be satisfied in studies that use quasi-
tal study into data that provide a firm and legitimate experimental designs, and highly unlikely to be sat-
basis for inferences about cause. Experimental isfied in studies that use nonexperimental designs.
design serves as the basis for inferences about cause, Even if one relaxed the temporal precedence
whereas statistical methods allow for inferences requirement, the validity of causal inferences using
about the existence of relations between variables data from nonexperimental studies would hinge on
and the probability that an observed estimate of the the assumed causal model being properly specified.
strength of a relation resulted from sampling error. Among the relevant specification considerations are
Note, moreover, that statistical methods are "blind" the proper ordering of variables in the model being
to the experimental design of the study that pro- tested, the inclusion of all important causes (exoge-
duced the data being analyzed. nous variables) and mediator variables in the model,
the correct specification of the form (e.g. .Iinear,
Causal Modeling nonlinear) of the relations considered by the model,
Although data from virtually all nonexperimental the correct specification of the causal paths in the
studies provide a very weak basis for making causal model (including those associated with reciprocal
inferences, a number of so-called causal modeling causation), and the use of measures of variables that
techniques are currently being used with such data to have high levels of construct validity. In this regard,
test assumed causal models, including PA, hierarchi- it is critical to understand that it is Virtually impossi-
cal multiple regression (HMR), hierarchical linear ble to satisfy these requirements in any nonexperi-
modeling (HLM), and SEM (Baron &:. Kenny, mental study (Bollen, 1989; Ling, 1982; Rogosa,
1986; Blalock, 1964, 1971; Bollen, 1989; Bryk &:. 1987; Rosopa &:. Stone-Romero, 2008; Stone-
Raudenbush, 1992; Cohen, Cohen, West, &:. Aiken, Romero, 2009; Stone-Romero &:. Rosopa, 2004;
2003; Kenny, 1979; Maruyama, 1998; Millsap, 2002). 2008). As a consequence, inferences about the valid-
A large number of examples of such analyses can be ity of assumed causal models that are based on such
found in articles published in the major journals in research rest on a very weak empirical foundation.
I/O psychology and allied disciplines (e.g., Academy As such, they are almost never justified. In this
ofManagementJoumal, Personnel Psychology,Joumal regard, Cliff (1987) noted that when a researcher is
of Applied Psychology, and Organizational Behavior analyzing data from nonexperimental research, "it is

60
 Research Strategies in I/O Psychology

not possible to isolate the empirical system suffi- (a)

ciently so that the nature of the relations among vari-
ables can be unambiguously ascertained" (p. 119).
Consistent with this view, Rogosa (1987) wrote that
"attempts to answer experimental ... research ques-
x
'I,---Y_
tions with nonexperimental data seem fundamentally
(b)
askew" (pp. 193-194). And, in accordance with these
arguments, Ling (1982) argued that "causal infer-
encelsl from correlational [i.e., nonexperimental]
data (in the absence of controlled experiments) ...
Y
'I_x_
are at best a form of statistical fantasy" (p. 490).

Model-Data Consistency Versus

(c)
x
Model-Reality Consistency
It is quite common for researchers who use causal
modeling procedures to argue that they have shown
z
support for an assumed causal model as a result of
model-data consistency, that is, showing consistency Y
between it and the covariances among variables con- '.'f
sidered by a nonexperimental study. The model-
FIGURE 2.4. Alternative causal models that are
data consistency issue is considered in relation to
consistent with an X-Y correlation. From Handbook
Figures 2.4a-2.4c, which show several possible of Organizational Research Methods (p. 319), by
causal models that are consistent with an observed D. Buchanan and A. Bryman (Eds.), 2009. London:
correlation between variables X and Y. Sage. Copyright 2009 by E. Stene-Romero. Reprinted
with permission.
Researchers who test assumed causal models
using data from nonexperimental research typically
advance three claims:
including (a) the model shown in Figure 2.4b,
1. The premise: If my assumed causal model
which posits that Y ~ X, and (b) the model shown
(e.g., that shown in Figure 2.4a) is correct, then
in Figure 2.4c, which specifies that Z ~ X and
the findings of my nonexperimental study will
Z ~ Y,implying that the correlation between X
be consistent with it.
and Yis noncausal. Assuming that the other models
2. The research findings: The findings of my study
(see Figures 2.4b and 2.4c) fit the data as well as the
are consistent with my assumed causal model; that
researcher's assumed causal model (see Figure 2.4a),
is, there is a nonzero relation between X and Y.
the researcher would have no firm basis for conclud-
3. The conclusion: Therefore, my assumed causal
ing that his or her preferred model is superior to
model is correct.
the others.
Unfortunately, this conclusion is not based on However, to buttress the argument that his or her
sound reasoning. More specifically, it is an instance assumed causal model is correct, the researcher may
of the logical fallacy of affirming the consequent argue that it is theory consistent. Unfortunately,
(Kalish &: Montague, 1964; Storie-Romero, 2009; this is of little or no value in terms of supporting
Stone-Romero &: Rosopa, 2004, 2008). For instance, claims about cause. The reason for this is that a
arguing that a theoretical model is correct because number of different theories may be used to
research results are consistent with the model. explain any observed covariance. Stone-Romero
It is easy to see why the just-noted conclusion is and Rosopa (2004, 2008) and Storie-Romero
unwarranted. It is that the findings also may be con- (2009) provided several illustrations of this. One
sistent with a number of other possible causal models, has to do with the relation between job satisfaction

61
 Eugene F. Stone-Romero

and job performance. Researchers associated with that motivation mediates the relation between
the human relations movement in VO psychology incentives and task performance. In Experiment 1
have argued that job satisfaction causes job perfor- incentives can be varied experimentally to assess
mance (e.g., see Pinder, 1998). However, expectancy their impact on both motivation and task perfor-
theorists view job satisfaction as a consequence of mance, and in Experiment 2 motivation can be
job performance (e.g., Porter &. Lawler, 1968). Still manipulated to determine its effect on task per-
others have posited that job satisfaction and job per- formance. Taken together, the results of these two
formance are reciprocally related to one another. experiments, along with reasoning from symbolic
Given these competing views, the researcher is in no logic (see Theorem 26 in Kalish &. Montague,
position to argue that one theory is more supported 1964), would provide strong support for the infer-
by the data than another. ence that M mediates the effect of X on Y.
A second example of competing theoretical
explanations concerns the relation between job Prediction Versus Causation
satisfaction and organizational commitment. One Tests of assumed causal models often lead to results
group of researchers has argued that organizational that can be used for prediction purposes. Here, the
commitment causes job satisfaction (e.g., Bateman term prediction is used in a statistical sense (Guion,
&. Strasser, 1984; Koslowsky, 1991; Lance, 1991; 1976, 1991,2002; Hays, 1994; Pedhazur, 1987).
Vandenberg &. Lance, 1992; Wiener &. Vardi, 1980). That is, information on a set of predictor variables
Another set has asserted that job satisfaction causes can be used to predict the value of a criterionff
organizational commitment (e.g., Williams &. interest. The fact that such prediction is possible
Hazer, 1986). Yet another group has contended that does not imply that the predictor is a cause ofthe
job satisfaction and organizational commitment are predicted variable (Bollen, 1989; Brannick, 1995;
reciprocally related to one another. Finally, some Cliff, 1987; Freedman, 1987; Holland, 1986;
researchers believe that the relation between these Kelloway, 1998; Ling, 1982; Mathieu et al., 2008;
two variables is spurious (e.g., Currivan, 1999). Millsap, 2002; Rogosa, 1987; Rosopa &. Stone-
Thus, a researcher who found a positive relation Romero, 2008; Stone-Remere, 2009; Stone-Romero
between these two variables would be hard-pressed &. Rosopa, 2004, 2008). For example, (a) the zip
to claim that his or her theoretical stance was supe- codes of individuals can be used to predict their
rior to numerous alternatives. annual income levels, (b) the weights of individuals
Overall, it should be clear that the invocation of a can be used to predict their heights, and (c) the.
theory is of little or no value in supporting the infer- heights of individuals at age 10 years can be used to
ence that the findings of a nonexperimental study predict their heights at age 30 years. In each of these
have internal validity. Thus, theories are of little or instances, the fact that the first variable can be used
no importance in terms of buttressing claims about to predict the second does not serve as a valid basis
the correctness of an assumed causal model for such for inferences about cause. Indeed, one can use sta-
a study. Clearly, the best strategy for showing sup- tistical methods for postdictive purposes; that is, a
port for such a model is to perform one or more researcher can use data from a currently measured
randomized experiments that test model-based variable to postdict events that took place in the past
predictions. Stene-Romero and Rosopa (2008) (Blum &. Naylor, 1968; Storie-Romero, 2007a). For
explained how such experiments can be used to example, current assessments of neurological dam-
support causal inferences for a simple causal model age in Viet Nam veterans can be used to postdict
in which the relation between X and Y is mediated their previous level of exposure to neurotoxins
-~--------hbrM-;-tha-t-is,-X --7 M--7 Y.More specifically, they (e.g., }\gentQra~ge).
~~--~~~~~~~~
argued that one randomized experiment can be used The very important implication is that the ability
to support the conclusion that X causes both M and to predict, in a statistical sense, has no necessary
Y, and a second can be used to show that M causes Y. implications for the understanding of causal
I
I Consider, for instance, research aimed at showing processes. Regrettably, there is considerable evi-

I
,I 62
.~
..i

!l
 Research Strategies in I/O Psychology

dence of the fact that many researchers don't under- tle more than surrogates for other nonmanipulable
stand this fact. This issue is considered in the sub- variables. For example, biological sex is a "stand-in"
section titled Unwarranted Causal Inference in for variables that differ between men and women,
Organizational Research. including values, motives, attitudes, aptitudes, and
abilities. Thus, in such research, it would be far bet-
Causal Inferences Associated With ter to measure such variables directly, as opposed to.
Nonmanipulable Variables measuring the surrogate of sex.
In randomized experimental studies, researchers
actually can manipulate such characteristics of stim- Unwarranted Causal Inference
ulus people as their age, race, biological sex, and in Organizational Research
ethnicity (e.g., via resume-based manipulations). As As noted previously, the appropriateness of causal
such, they can test for the effects of such variables inferences varies as a function of a study'S experimen-
on measured outcomes (e.g., hiring decisions). As a tal design. Unfortunately, however, inappropriate
result, they can support inferences about the impact inferences about causal connections between vari-
of such variables on the outcomes. However, in non- ables (i.e., inferences that are not justified by the type
experimental studies, the same variables are mea- of experimental design of a study) are quite common
sured, as opposed to manipulated, and are often in reports of the findings of nonexperimental studies.
assumed to have a number of roles. One of these is Unwarranted causal language (e.g., the use of
that of a moderator (Storie-Romero &:. Liakhovitski, such terms as causes, effects, and influences) can be
2002; Zedeck, 1971). For instance, in criterion- found in very large percentages of the articles p~b~
related validation studies, race has been studied as a lished in the major journals of I/O psychology and
moderator of the validity of predictors of job perfor- allied disciplines. Evidence of this comes frorn'a
mance (e.g., Boehm, 1972; Schmidt &:. Hunter, study by Storie-Romero and Gallaher (2006). They
1974). A second role is that of an assumed indepen- content analyzed 161 randomly sampled articles
dent variable. For example, numerous studies have that were published in the 1988, 1993, 1998, and
examined the relation between the stigmas of targets 2003 volumes of Personnel Psychology,
and their treatment by others (e.g., Stene-Romero &:. Organizational Behavior and Human Decision
Stone, 2007). Third, and finally, such variables are Processes, the Academy of ManagementJournal, and
often used as controls in statistical analyses. It is the Journal of Applied Psychology. The articles
interesting, however, that that researchers typically reported the findings of studies that used random:
fail to provide a sound rationale for this use of non- ized experimental, quasi-experimental, and non-
manipulable variables. Even if they do, statistical experimental designs. Storie-Romero and Gallaher
controls are seldom effective (Storie-Romero, 2007a). searched the same publications for instances of the
There are numerous reasons for this, one of which inappropriate use of causal language in their title,
is that when control variables are measured in an abstract, and results and/or discussion sections. The
unreliable manner, their "effects" are not fully analysis revealed that unwarranted causal language
removed by partialling procedures (e.g., calculation appeared one or more times in 58 of the 73 articles
of partial regression coefficients in multiple regres- that reported the results of research that used non-
sion analyses). experimental designs (79% of such articles) and
An important issue in nonexperimental studies is 14 of the 18 articles that reported the findings of
the degree to which researchers can make valid research that used quasi-experimental designs (78%
inferences about the effects of nonmanipulable vari- of such articles). For examples of articles that con-
ables. In view of internal validity considerations, it is tain one or more instances of unwarranted causal
inappropriate to argue about the effects of various inference, see Storie-Romero and Rosopa (2008) or
nonmanipulable variables. It deserves adding that Storie-Romero (2009).
the vast majority of nonmanipulable variables that Overall, what the studies considered by Stone-
are studied in nonexperimental studies represent lit- Romero and Gallaher (2006) show is that researchers .

63
 Eugene F. Stene-Romero

often make unwarranted claims about cause on the was much higher in management journals (reviewed
basis of studies that do not provide a basis for such by Scandura &. Williams, 2000) than in psychology
claims. In addition, they often assume, quite incor- journals (reviewed by Austin et al., 2002, and Stone-
rectly, that data analysis, as opposed to sound experi- Romero et al., 1995).
mental design, affords an adequate foundation for The reasons for the variability in the types of
advancing causal inferences. Clearly, if researchers experimental designs used by researchers across
are interested in making such inferences, they must time are unclear. However, two possibilities merit
conduct research using randomized experimental or consideration. One is that they were at least in part a
sound quasi-experimental designs. function of the types of designs that- were favored by
the editors and editorial board members of the jour-
TRENDS IN RESEARCH DESIGN nals considered by the reviews. For instance, the
AND METHODS journals considered by the Scandura and Williams
(2000) review are less likely to publish the results of
There are a number of notable trends in the designs experimental studies than is the joumal of Applied
used in research in I/O psychology and related fields Psychology. Another possibility is that the issues
that are detailed in reviews by Austin et al. (2002), addressed by researchers may have influenced the
Scandura and Williams (2000), and Storie-Romero experimental designs that they used. fur example,
et al. (1995). In addition, there is evidence on the some topics (e.g., goal setting, training) are more
types of articles that have been published in a jour- amenable to study by experimental means than
nal that focuses on methods-related issues (Aguinis are others (e.g., organizational commitment, job
et al., 2009). These reviews are considered in the satisfaction). "-
following section, and their major findings are Whatever the reason(s) for the predominant
summarized in Table 2.1. use of nonexperimental designs in organizational
research, it seems clear that greater use must be
Design Trends made of both quasi-experimental and randomized
The Austin et al. (2002) study considered 609 articles experimental designs. Consistent with this view,
sampled from nine volumes of the Journal of Applied Highhouse (2009) argued that "despitethe benefits
Psychology in lO-year intervals starting at 1920 and of randomized experimentation ... for making
ending at 2000. The Stone-Romero et al. (1995) causal inferences, organizational scholarship has
review was based on 1,929 articles published in the historically been characterized by an unhealthy
same journal for each of 19 years (1975-1993). over-reliance on [research based on] passive obser-
Finally, the Scandura and Williams (2000) article vation [i.e., nonexperimental designs]" (p ..554).
dealt with all articles (N = 774) published in three
Research settings. Neither Scandura and
management-oriented journals (i.e., Academy of
Williams (2000) nor Stene-Romero et al. (1995)
ManagementJournal, Administrative ScienCe Quarterly,
I I provided information on the settings of the studies
and Journal of Management) for the 1985-1987 and
considered by their review. However, Austin et al.
1995-1997 periods. A brief summary of their findings
(2002) reported that the settings in which research
follows.
has been conducted have varied from one decade to
Experimental design. Research shows that the another. In contrast to the SP versus NSP setting dis-
use of randomized experimental, quasi-experimental, tinction explained previously, Austin et al. (2002)
and nonexperimental designs has fluctuated some- distinguished between laboratory and field settings:'
what over time. Nonexperimental designs were used Thus, the same distinction is used here. They
quite frequently in the periods considered by the reported that the pas conducted in (a) laboratory
reviews. In fact, they are the most frequently used settings varied from 19.3 (1940) to 41.5 (1980) and
type of design. Note, moreover, that the percentage was 22.4 in 2000, (b) field settings ranged from 53.7
of studies (PaS) that used nonexperimental designs (1980) to 80.7 (1940) and was 65.7 in 2000, and

64
 TABLE 2.1

Trends in Research: Percentage of Studies With Various Design and Analysis-Related Attributes

Findings from review of

Austin et al. (2002) Scandura & Williams (2000) Stone-Romero et al, (1995)
Study attribute Low (year) High (year) Low (year) High (year) Low (year) High (year)
Experimental design
Experimental 14.6 (1930) 52.9 (1970) 15.5 (1985-1987) 10.0 (1995-1997) 32.5 (1990) 43.88 (1993)
Nonexperimental 43.9 (1980) 78.0 (1930) 61.6 (1985-1987) 90.0 (1995-1997) 36.73 (1993) 58.67 (1975)
Research setting
Special purpose 19.3 (1940) 41.5 (1980)
Nonspecial purpose 53.7 (1980) 80.7 (1940)
Simulation 0.0 (1920) 6.8 (1990)
Data analysis technique
ANOVAa 0.0 (1920) 51.2 (1980) 4.00 (1975) 13.64 (1990)
MANOVA 0.0 (1920) 16.2 (1990) 2.34 (1977) 12.24 (1993)
ANCOVA 0.0 (1920) 4.5 (2000) 0.67 (1975) 6.59 (1992)
MANCOVA 0.0 (1920) 2.7 (1990) 0.0 (1975) 2.56 (1983)
Multiple regression and variants' 1.8 (1940) 46.3 (2000) 30.70 (1985-1987) 42.40 (1995-1997) 10.16 (1977) 32.98 (1988)
Factor analysis (exploratory)' 0.0 (1920) 9.8 (1980) 8.3 (1985-1987) 5.10 (1995-1997)d 2.35 (1979) 16.04 (1982)
SEM
Confirmatory factor analysis 0.0 (1920) 16.4 (2000) 4.0 (1985-1987) 6.60 (1995-1997) 0.0 (1975) 8.16 (1993)
Path analysis/structural model 0.0 (1920) 3.0 (2000) 0.0 (1975) 6.82 (1990)
Latent variable SEM' 0.0 (1920) 13.5 (1990) 3.6 (1985-1987) 6.90 (1995-1997) 0.0 (1975) 10.20 (1993)

Note. ANOVA = analysis of variance; MANOVA = multivariate analysis of variance; ANCOVA = analysis of covariance; MANCOVA = multivariate analysis of covariance;
SEM = structural equation modeling. Dashes indicate that no information on this variable was reported in the review.
"Includes between-subjects, within-subjects, and mixed designs.
blncludes ordinary least squares; weighted least squares, and logistic.
-Includes principal components analysis and common factor analysis.
dFor Scandura and Williams review, category includes factor analysis and clustering techniques.
"For Scandura and Williams review, category includes SEM and path analysis.
 Eugene F. Stone-Romero

(c) simulation settings extended from 0.0 (1920) to for testing SEM models is now part of the widely
6.8 and was 3.0 in 2000. Clearly, field settings are available SPSS software package.
the most commonly used in 110 research.
Methods-Related Articles
Data analytic strategies. A review of the articles
Aguinis et a1. (2009) reported the results of a content-
by Austin et a1. (2002), Scandura and Williams
analysis-based review of 193 articles that were pub-
(2000), and Stene-Romero et a1. (1995) shows that
lished in Organizational Research Methods from 1998 .
there are notable trends in the data analytic strategies
to 2007. Among the findings were that the percentage
used by researchers. Although a thorough considera-
of articles (POAs) focusing on quantitative methods
tion of all of them is beyond the scope of this chapter,
were much greater than those for qualitative methods. '
the major ones are as follows.
In addition, for quantitative methods, there were
1. ANOVA variants. First, the use of multivariate (a) increases in the POAs focusing on electronic!Web
data analytic strategies that are variants of research, survey research, and multilevel data analytic
ANOVA (e.g., multivariate analysis of variance methods, and (b) decreases in the POAs dealing with'
[MANOVA], analysis of covariance, and multi- SEM. Moreover, for qualitative methods there were
variate analysis of covariance) has increased (a) increases in the POAs devoted to action research
considerably. Consider, for instance, the use of and interpretive research and (b) increases
MANOVA: The reviews by both Storie-Romero (1998-2002) followed by decreases (2002-2006) in
et a1. (1995) and Austin et a1. (2002) reported the POAs concerned with policy capturing. These
substantial increases in the POS using this tech- trends may suggest methods-related education and
nique. Scandura and Williams (2000) did not training needs for basic and researchers in 110 psy-
provide information on its use. chology and allied fields. However, care must be exer-
2. Multiple regression-based techniques. All three cised in interpreting them. More specifically, articles
reviews indicated that there were marked on a specific method may focus on explanations of i~
increases in the use of multiple regression-based proper use, controversies surrounding its use, or criti-'
data analytic strategies. The largest such increase cal views on its use. Consider three examples of the
was reported by Austin et a1. (2002). It appears latter possibility: Rogosa (1987) described the erro-
to be a result of the rather long time period con- neous inferences that can result from the use of cross-
sidered by their study (i.e., 1920 to 2000). lagged panel correlation strategy (e.g., Kenny, 1975,-'
3. Structural equation modeling. The use of SEM 1979) for deriving evidence about causal relations
techniques has increased greatly in the periods using data from nonexperimental research: Kalleberg
covered by the three reviews. This is true for and Kleugel (1975) pointed out the problems that
analyses aimed at testing (a) measurement can result from the use of the MTMM strategy for
models (i.e., CFA), (b) SEM or path models, and assessing the construct validity of measures. Stone-
(c) latent variable structural equation models Romero and Anderson (1994) showed why moder-
(LVSEM). These increases appear to be a result of ated multiple regression is a better str~tegy for
several factors. First, training in many SEM meth- detecting moderating effects than testing for differ-
ods (covariance structure analysis, path analysis, ences among subgroup-based correlation coefficients.
and LVSEM) procedures was not widely available In short, publications on a given topic are not always
i'
to researchers in 110 psychology and related dis- a signal that it is of value to researchers.
ciplines until the 1970s. For instance, the first
article on CFA appeared in 1969. Second, the
CONCLUSIONS
software needed to conduct SEM analyses (i.e.,
LISREL 3) did not appear until 1976 and was Several conclusions appear warranted on the basis of
I
I both expensive and difficult to use. Today, how- the foregoing. They have to do with the implications
I
;,j
"J I ever, SEM software is readily available and quite of research design for validity inferences, the dis-
user friendly. For example, the AMOS software tinction between experimental design and data ana-

66
 Research Strategies in IIO Psychology

lytic strategies, and the value of methodological inflated study-wise Type I error. In addition, the
pluralism in empirical research. study would be of no value if there truly is a relation
between the variables, but the study lacks the statis-
Research Design and Validity Inferences tical power to detect it.
A researcher has many options when designing a Third, and finally, a study might be very strong
study, including those that pertain to its experi- in terms of internal validity, construct validity, and
mental design, which is the most important deter- statistical conclusion validity, but weak with regard
minant of its internal validity. As noted previously, to external validity. Thus, if it is important to make
the experimental design used in a study should be inferences about a causal relation to sampling par-
governed by its major purposets). If the sole inter- ticulars other than those considered by a study
est of a researcher is to determine if two or more (e.g., to other sets of units, treatments, outcomes,
variables are related to one another, he or she can and settings), the researcher must conduct addi-
conduct a study that uses a nonexperimental tional studies to support claims about external
design. However, if the researcher is concerned validity. It is important to add, however, that exter-
with determining causal relations between or nal validity is not always an important concern in
among variables, he or she should opt for a ran- research (Mook, 1983).
domized experimental study (Cook &: Campbell,
1976,1979; Cook et a1., 1990; Shadish et al., 2002; Experimental Design and Data
Stone-Romero, 2002, 2007c, 2009; Stene-Romero Analytic Strategies
&: Rosopa, 2004, 2008). And, in instances in which From the foregoing it should be clear that there is a
experimentation is possible but units cannot be ran- very important distinction between experimental
domly assigned to treatment conditions, a quasi- design and the methods used to analyze data from a
experimental study should be conducted (Shadish study. Regrettably, large num:Bers of publications in.
et a1.,2002; Stone-Remere, 2002, 2007b, 2009; l/O psychology and allied disciplines provide clear "
Stene-Romero &: Rosopa, 2008). evidence of a very prevalent problem: Researchers
Irrespective of the experimental design used in a who are interested in showing causal relations
study, a researcher must devote attention to the between (among) variables seem to think that statis- P

other facets of validity. First, to bolster inferences tical methods are an appropriate substitute for
about construct validity, he or she must take care to sound experimental design; that is, they base con-
ensure that the sampling particulars of a study (e.g., clusions about causal connections between (among)
units, treatments, settings, and outcomes) are faith- variables on the results of applying so-called causal
ful representations of associated constructs. It is of modeling procedures (e.g., HMR, PA, SEM) to data
virtually no value to show that a manipulation of X from nonexperimental research (see Stene-Romero,
leads to changes in Yif a researcher has little or no 2009; Stone-Romero &: Gallaher, 2006;2,tone-
knowledge about the constructs associated with Romero &: Rosopa, 2004, 2008). However, valid
these variables. Thus, for example, the construct inferences about cause stem from sound experimen-
validity of a study would be questionable if it dealt tal design, not the use of so-called causal modeling
with the decision-making processes of neurosur- procedures. Consequently, when nonexperimental
geons but used undergraduate business students as research is reported, researchers must be careful to
subjects. avoid causal interpretations of their findings. Recall,
Second, a researcher has to be concerned about for example, that an observed relation between two
the validity of inferences derived from statistical variables (e.g., X and Y) can be consistent with mul-
analyses, that is, statistical conclusion validity. It is tiple assumed causal models (see Figure 2.4), and a
of almost no value for a researcher to claim that a researcher typically has no basis for legitimately
study shows support for a hypothesized relation arguing that one such model (e.g., Figure 2.4a) is
between two variables if this finding is based on a superior to the alternatives (e.g., Figures 2.4b
study that suffers, for example, from the problem of and 2.4c).

67
r"··
Eugene F Stone-Romero

I
!
! Methodological Pluralism construct, external, and statistical conclusion valid-
In studies of virtually all phenomena in I/O psychol- ity. However, of these types of validity, internal is the
ogy and related disciplines, methodological pluralism sine qua non (J. C. Campbell &: Stanley, 1963).
is highly desirable (Rosenthal &: Rosnow, 2008;
Shadish et al., 2002; Webb et al., 1981). The call for References
methodological pluralism is consistent with the fiat Aguinis, H., Pierce, C. A., Bosco, F. A., & Muslin, I. S.
mistura (i.e., let there be a mixture) prescription of (2009). First decade of Organizational Research.
Methods: Trends in design, measurement, and data
Fromkin and Streufert (1976). In early stages of
analysis topics. Organizational Research Methods,
research on a phenomenon, nonexperimental studies 12,69-112.
may offer valuable clues about the correlates of a vari- Ajzen, I. (1988). Attitudes, personality, and behavior.
able that is of interest to an investigator. In addition, Chicago: Dorsey Press.
the results of nonexperimental research may provide Argyris, C. (1968). Some unintended consequences of rig-
a basis for the formulation of hypotheses about causal orous research. Psychological I?ulletin, 70, 185-197.
connections between the same variable and its Argyris, C. (1980). Inner contradictions of rigorous
assumed antecedents. However, such hypotheses are research. New York: Academic Press.
best tested using either randomized experimental or Aronson, E., Ellsworth, P., Carlsmith,]. M., & Gonzales,
appropriate quasi-experimental designs. M. H. (1990). Methods of research in social psychology
(2nd ed.), New York: McGraw-Hill.
Another highly important dimension of method-
Austin,]. T., Scherbaum, C. A., & Mahlman, R. A.
ological pluralism relates to the empirical realiza-
(2002). History of research methods in industrial
tions of the constructs considered by a study. In any and organizational psychology: Measurement,
given study, it is vital to demonstrate that there is a design, analysis. In S. G. Rogelberg (Ed.), Handbook
clear connection between any given construct and of research methods in industrial and organizational
psychology (pp. 1-33). Malden, MA. ,mackwell.
its sampling particulars (e.g., manipulations ancIJor
Baron, R. M., & Kenny, D. A. (1986). The moderator-
measures of variables). However, in subsequent
mediator variable distinction in social psychological
studies it is critical to demonstrate that the observed research: Conceptual, strategic, and.statistical con-
relation between studied variables is not an artifact siderations. Journal of Personality and Social
of sampling particulars. Thus, for example, if a Psychology, 51, 1173-1182.
researcher used a specific measure of an assumed Bateman, T. S., & Strasser, S. (1984). A longitudinal
analysis of the antecedents of organizational commit-
effect construct in one study, he or she should con-
ment. Academy of Management Journal, 27, 95-i12.
duct follow-up research that uses alternative mea-
Berkowitz, l., & Donnerstein, E. (1982). External validity
sures of it. In addition, if the researcher manipulated is more than skin deep: Some answers to criticisms of
an independent variable in a particular way in one laboratory experiments. American Psychologist, 37,
study, he or she should use alternate manipulations 245-257. '~
in subsequent studies. Blalock, H. M. (1964). Causal inferences in nonexperimen-
tal research. New York: W. W. Norton.
A Final Word Blalock, H. M. (1971). Causal models in the social sciences:
Chicago: Aldine.
Sound research is vital to developing an understand-
ing of causal relations between focal variables and Blanchard, E.B., Peters, M.l., Hermann, c., Turner, S. M.,
Buckley, T. c, Barton, 1\" & Dentinger, M. P. (1997).
their antecedents and consequences. The findings of Direction of temperature control in the thermal
such research can (a) contribute greatly to the devel- biofeedback treatment of vascular headache. Applied
opment and testing of theories and (b) provide a firm Psychophysiology and Biofeedback, 22, 227-245.
basis for the design of interventions aimed at chang- Blum, M., & Naylor,]. C. (1968.). Industrial psychology:
ing individuals, groups, and organizations. Thus, in Its theoretical and social foundations. New York:
Harper and Row.
terms of multiple criteria that are highly important to
Boehm, V. R. (1972). Negro-white differences in validity
both scientists and practitioners in I/O psychology
/Z.' of employment and training selection procedures:
and related n,€lds, it is critical that research on rela- Summary of research evidence. Journal of Applied
tions between variables have high levels of internal, Psychology, 56, 33-39.

68
 Research Strategies in 110 Psychology

Bollen, K. A. (1989). Struct~ral equations with latent Cohen,]., Cohen, P.,West, S. G., &:. Aiken, L. S. (2003).
variables. New York: WIley. Applied multiple regression!correlation analysis for the
behavioral sciences (3rd ed.). Mahwah, NJ: Erlbaum.
Borm an , W .. C (1991).Job behavior, performance, and
effectiveness. In M. D. Dunnette &:. L. M. Hough Conway,]. M. (2002). Method variance and method
(Eds.), Handbook of industrial and organizational psy- bias in industrial and organizational psychology.
chology (2nd ed., Vol. 2, pp. 271-326). Palo Alto, In S. G. Rogelberg (Ed.), Handbook of research
CA: Consulting Psychologists Press. methods in industrial and organizational psychology
B tt J. P. (2007). Case study method. In S. G. Rogelberg (pp. 344-365). Malden, MA: Blackwell.
o '(Ed,), Encyclopedia of industrial and organizational Cook, T. D., &:. Campbell, D. T. (1976). The design and
psychology (Vol. 1, pp. 68-69). Thousand Oaks, CA: conduct of quasi-experiments and true experiments
Sage. in field settings. In M. D. Dunnette (Ed.), Handbook
Bouchard, T. (1976). Field research methods: Interviewing, of industrial and organizational psychology
questionnaires, participan~ observation, systematic (pp. 223-326). Chicago: Rand McNally.
observation, and unobtrusive measures. In M. D. Cook, T. D., &:. Campbell, D. T. (1979). Quasi-
Dunnette (Ed.), Handbook of industrial and organiza- experimentation: Design and analysis issues for
tional psychology (pp. 363-413). Chicago: Rand field settings. Boston: Houghton Mifflin.
McNally.
Cook, T. D., Campbell, D. T., &:. Peracchio, L. (1990).
Boudreau,J W. (1991). Utility analysis for decisions in Quasi-experimentation. In M.; D. Dunnette &:. L. M.
human resource management. In M. D. Dunnette &:.
Hough (Eds.), Handbook of industrial and organiza-
L. M. Hough (Eds.), Handbook of industrial and orga-
tional psychology (2nd ed., Vol. 1, pp. 491-576). Palo
nizational psychology (2nd ed., Vol. 2, pp. 621-717).
Alto, CA: Consulting Psychologists Press.
Palo Alto, CA: Consulting Psychologists Press.
Cronbach, L.]., &:. GIeser, G. C. (1965). Psychological
Braithwaite, R. (1996). Scientific explanation. Herndon, .
tests and personnel decisions. Urbana: University of
VA: Books International.
Illinois Press.
Brannick, M. T. (1995). Critical comments on applying
Cronbach, L.]., &:. Meehl, P. E. (1955). Construct validity
covariance structure modeling. Journal of
in psychological tests. PsycholOgical Bulletin, 52,
Organizational Behavior, 16, 201-213.
281-302.
Bryk, A. S., sx Raudenbush, S. W. (1992). Hierarchical
Currivan, D. B. (1999). The causal order of job satisfac-
linear models: Applications and data analysis methods.
tion and organizational commitment in models of
Newbury Park, CA: Sage.
employee turnover. Human Resource Management
Buchanan, D., &:. Bryman, A. (Eds.). (2009). Handbook of Review, 9, 495-524.
organizational research methods. London: Sage.
Dipboye, R. L., ts: Flanagan, M. F. (1979). Research set-
Campbell, D. T., &:. Fiske, D. (1959). Convergent and dis- tings in industrial and organizational psychology:
criminant validation by the multitrait-multimethod Are findings in the field more generalizable than in
matrix. Psychological Bulletin, 56, 81-105. the laboratory? American Psychologist, 34, 141-150. J

Campbell, D. T., &:. Stanley,]. C. (1963). Experimental Doty, D. H., &:. Glick, W. H. (1998). Common method
and quaSi-experimental designs for research. Chicago: bias: Does common methods variance really bias
Rand McNally. results? Organizational Research Methods, 1, 374-406.
Campbell,]. P. (1976). Psychometric theory. In M. D. Feldman,]. M., &:. Lynch,]. G. (1988). Self-generated
Dunnette (Ed.), Handbook of industrial and organiza- validity and other effects of measurement on belief,
tional psychology (pp. 185-222). Chicago: Rand attitude , intention, and behavior. Journal of Applied
.
McNally.
Psychology, 73,421-435.
Campbell,]. P. (1986). Labs, fields, and straw issues. In
Freedman, D. A. (1987). As others see us: A case study in
E. A. Locke (Ed.), GeneraliZingfrom laboratory to field path analysis. Journal of Educational Statistics, 12,
settings: Research findings from industrial-organizational 101-128.
psychology, organizational behavior, and human
resource management (pp. 269-279). Lexington, MA; Frese, M., Garst, H., &:. Fay, D. (2007). Making things
Lexington Books. happen: Reciprocal relationships between work char-
acteristics and personal initiative in a four-way Iongi-
Cliff, N. (1987). Comments on Professor Freedman's
tudinai structural equation model. Journal of Applied
paper. Journal of Educational Statistics, 12, 158-160.
Psychology, 92, 1084-1102.
Cochran, W. G. (1977). Sampling techniques (3rd ed.).
Fromkin, H. L., &:. Streufert, S. (1976). Laboratory:
New York: Wiley.
experimentation. In M. D. Dunnette (Ed.), Handbook
Coh~n, J. (1988). Statistical power analysis for the behav- of industrial and organizational psychology ,
IOral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. (pp. ~ 15-465). Chicago: Rand McNally.

69
Eugene F. Stone-Romero

Ghiselli, E. E. (1956). Dimensional problems of criteria. Kenny, D. A. (1975). Cross-lagged panel correlation: A test
Journal of Applied Psychology, 40, 1-4. for spuriousness. Psychological Bulletin, 82, 887-903.
Ghiselli, E. E., Campbell,]. P., &: Zedeck, S. (1981). Kenny, D. A. (1979). Correlation and causality. New York:
Measurement theory for the behavioral sciences. New Wiley.
York: W. H. Freeman.
Kerlinger, F. N., &: Lee, H. B. (2000). Foundations of
Grant, A. M., &: Wall, T. D. (in press). The neglected behavioral research (4th ed.). Belmont, CA:
science and art of quasi-experimentation: Why-to, Wadsworth.
when-to, and how-to advice for organizational
researchers. Organizational Research Methods. Kish, L (1965). Survey sampling. New York: Wiley.

Guion, R. M. (1976). Recruiting, selection, and job place- Koslowsky, M. (1991). A longitudinal analysis of job
ment. In M. D. Dunnette (Ed.), Handbook of indus- satisfaction, commitment, and intention to leave.
trial and organizational psychology (pp. 777-828). Applied Psychology: An International Review, 40,
Chicago: Rand McNally. 405-415.

Guion, R. M. (1991). Personnel assessment, selection, Kuhn, T. S. (1970). The structure of scientific revolutions
and placement. In M. D. Dunnette &: L M. Hough (2nd ed., enlarged). Chicago: University of Chicago
(Eds.), Handbook of industrial and organizational psy- Press.
chology (2nd ed., Vol. 2, pp. 327-397). Palo Alto, Lance, C. E. (1991). Evaluation of a structural model
CA: Consulting Psychologists Press. relating job satisfaction, organizational commitment,
Guion, R M. (2002). Validity and reliability. In S. G. and precursors to voluntary turnover.Multivariate
Rogelberg (Ed.), Handbook of research methods in Behavioral Research, 26, 137-162.
industrial and organizational psychology (pp. 57-76). Lance, C. E., Hoffman, B.]., Gentry, W. A., &: Baranik,
Malden, MA: Blackwell. L E. (2008). Rater source factors represent impor-
Hackman,]. R., &: Oldham, G. R. (1976). Motivation tant subcomponents of the criterion construct space,
through the design of work: Test of a theory. not rater bias. Human Resource Management Review,
Organizational Behavior and Human Performance, 16, 18, 223-232.
250-279. Lawler, E. E., Mohrman, A. M., Mohrman, S. A., Ledford,
Hackman,]. R, Pearce,]. L, &: Wolfe,]. C. (1978). Effects G. E., Cummings, T. G., &: Associates. (1985). Doing
of changes in job characteristics on work attitudes and research that is useful for theory and practic"e. San
behaviors: Anaturally occurring quasi-experiment. Francisco: jossey-Bass,
Organizational Behavior and Human Performance, 21, Ling, R. F. (1982). Review of Correlation and causality by
289-304. David A. Kenny. Journal of the American Statistical
Harris, M., &: Schaubroeck,]. (1988). A meta-analysis Association, 77, 489-49l. . '"
of self-supervisory, self-peer, and peer-supervisory
Locke, E. A. (1986). GeneraliZingfrom laboratory to
ratings. Personnel Psychology, 41, 43-62.
field settings: Research findings from industrial-
Hays, W. L (1994). Statistics (5th ed.). Fort Worth, TX: organizational psychology, organizationalDehavior,
Harcourt Brace. and human resource management. Lexington, MA:
Hedges, LV., &: Olkin, I. (1985). Statistical methods for Lexington Books.
meta-analysis. New York: Academic Press. Locke, K, &: Golden-Biddle, K (2002). An introduction
Highhouse, S. (2009). Designing experiments that gener- to qualitative research: Its potential for industrial and
alize. Organizational Research Methods, 12, 554-566. organizational psychology. In S. G. Rogelberg (Ed.),
Handbook of research methods in industrial and organi-
Holland, P. W. (1986). Statistics and causal inference. zational psychology (pp. 99-118). Malden, MA:
Journal of the American Statistical Association, 81, Blackwell.
945"':960.
Maruyama, G. M. (1998). Basics of structural equation
Kalish, D., &: Montague, R. (1964). Logic: Techniques offor- modeling. Thousand Oaks, CA: Sage.
mal reasoning. New York: Harcourt, Brace, &: World.
Mathieu.]. E., DeShon, R P., &: Bergh, D.·D. (2008).
Kalleberg, A. L, &: Kluegel,]. R. (1975). Analysis of the Mediational inferences in organizational research:
multitrait-multimethod matrix: Some limitations and
Then, now, and beyond. Organizational Research
an alternative. Journal of Applied Psychology, 60, 1-9.
Methods, 11, 203-223.
Kaplan,A. (1964). The conduct of inquiry. New York: Intext.
Mathieu,]. E., &: Taylor, S. R. (2006). Clarifying condi-
Kelloway, E. K (1998). Using LISRELfor structural equa- tions and choice points for mediational type infer-
tion modeling: A researcher's guide. Thousand Oaks, ences in organizational behavior. Journal of "
CA: Sage. Organizational Behavior, 27, 1031-1056.

70
 Research Strategies in 110 Psychology

McDonald, R P. (1999). Test theory: A unified treatment. Scandura, T. A., &. Williams, E. A. (2000). Research
Mahwah, NJ: Erlbaum. methodology in management: Current practices,
trends, and implications for future research. Academy
Millsap, R. E. (2002). Structural equation m?deling: A
of Management journal, 43, 1248-1264.
user's guide. In F. Drasgow &. N. Schmitt (Eds.),
Measuring and analyzing behavior in organizations Schmidt, F. L., &. Hunter,]' E. (1974). Racial and ethnic
(pp, 257-300. San Francisco: jossey-Bass. bias in psychological tests: Divergent implications
of two definitions of test bias. American Psychologist,
Mook, D. G. (1983). In defense of external invalidity. 29,1-8.
American Psychologist, 38, 379-387 ..
Searle, S. R (1971). Linear models. New York: Wiley.
Nunnally,]. C. (1978). Psychometric theory (2nd ed.).
New York: McGraw-Hill. Shadish, W. R, Cook, T. D., &. Campbell, D. T. (2002).
Experimental and quasi-experimental designs for gener-
Nunnally,]. c., &. Bernstein, 1. H. (1994). Psycaomeiru: . alized causal inference. Boston: Houghton Mifflin.
theory (3rd ed.). New York: McGraw-Hill.
Smith, P. C. (1976). Behaviors, results, and organiza-
Orne, M. T. (1962). On the social psychology of the psy- tional effectiveness: The problem of criteria. In M. D.
chological experiment: With particular reference to Dunnette (Ed.), Handbook of industrial and organiza-
demand characteristics and their implications. tional psychology (pp. 745-775). Chicago: Rand
Americ(ln Psychologist, 17, 776-783. McNally.
Orne, M. T. (1969). Demand characteristics and the Snedecor, G. W., &. Cochran, W. G. (1980). Statistical
concept of quasi-controls, In R. Rosenthal &. R. L. methods (7th ed.). Ames: Iowa State University. ,
Rosnow (Eds.), Artifact in behavioral research
Spector, P. (2006). Method variance in organizational
(pp. 143-179). New York: Academic Press.
research: Truth or urban legend? Organizational
Pedhazur, E.]. (1982). Multiple regression in behavioral Research Methods, 9,221-232.
research: Explanation and prediction (2nd ed.). New
Stanton,]. M., &. Rogelberg, S. G. (2002). Beyond online
York: Holt, Rinehart and Winston.
surveys: Internet research opportunities for industrial
Pinder, C. C. (1998). Work motivation in organizational and organizational psychology. In S. Rogelberg (Ed.),
behavior. Upper Saddle River, NJ: Prentice-Hall. The encyclopedia of industrial mid organizational psy-
Podsakoff, P. M., MacKenzie, S. B., Lee,]-Y., &. chology (pp. 275-2~4). Beverly Hills, CA: Sage.
Podsakoff, N. P. (2003). Common method biases in Stone, D. L., &: Stone, E. F. (1985). The effects offeed-
behavioral research: A critical review of the literature back favorability and feedback consistency on self-
and recommended remedies. journal of Applied perceived task competence and perceived feedback
Psychology, 88, 879-903. accuracy. Organizational Behavior and Human
Decision Processes, 36, 167-185.
Porter, L. W., &: Lawler, E. E. (1968). Managerial atti-
tudes and performance. Homewood, IL: Irwin-Dorsey. Stone, E. F. (1978). Research methods in organizational
behavior. Santa Monica, CA: Goodyear.
Rogosa, D. (1987). Causal models do not support scientific
conclusions: A comment in support of Freedman. Stone, E. F. (1979). Field independence and perceptions
journal of Educational Statistics, 12, 185-195. of task characteristics: A laboratory investigation.
journal oIApplied Psychology, 64, 305-310.
Rosenthal, R. (1991). Meta-analytic procedures for social
research (Rev. ed.). Newbury Park, CA: Sage. Stone, E. F. '(1981). Action science: Pulling the trigger on
research rigor. Contemporary Psychology, 26, 629-630.
Rosenthal, R, &. Rosnow, R. L. (1969). Artifact in behav-
ioral research. New York: Academic Press. Stone, E.F. (1982). In defense of rigorous research.
Contemporary Psychology, 27, 58l.
Rosenthal, R, &. Rosnow, R. L. (2008). Essentials ofbehav- (

ioral research (3rd ed.). New York: McGraw-Hill. Stone, E. F. (1987). Review of Doing research that is useful
for theory and practice, by E. E. Lawler et al.joumal
Rosopa, P.]., &. Stone-Romero, E. F. (2008). Problems of Occupational Behavior, 8, 87-9l.
with detecting assumed mediation using the hierar-
Stone, E. F. (1992). A critical analysis of social informa-
chical multiple regression strategy. Human Resource
Management Review, 18, 294-310. tion processing models of job perceptions and job
attitudes. In c.]. Cranny, P. C. Smith, &. E. F. Stone
Runkel, P.]., &. McGrath,]. E. (1972). Research on human (Eds.),job satisfaction: How people feel about their jobs
behavior: A systematic guide to method. New York: and how it affects their performance (pp. 21-44).
Holt, Rinehart, &. Winston. Lexington, MA: Lexington-Heath.
Salancik, G. R, &: Pfeffer,]. (1978). A social information Stone, E. F., Gueutal, H. G., Gardner, D. G., &: McClure, S.
proc~s~ing approach to job attitudes and task design. (1983). A field experiment comparing information-
AdminIstrative Science Quarterly, 23, 224-253. privacy values, beliefs, and attitudes across several

71
 Eugene F. Stone-Romero
1
types of organizations. Journal of Applied Psychology,
68, 459-468.
pedia of measurement and statistics (pp. 995-999).
I
1
Beverly Hills, CA: Sage. I

1
Stone, E. F., Stone, D. L., & Gueutal, H. G. (1990). The Storie-Romero, E. F., & Rosopa, P.]. (2008). The relative 1
influence of cognitive ability on responses to ques- validity of inferences about mediation as a function .~
,
tionnaire measures. Journal of Applied Psychology, 75,
418-427.
of research design characteristics. Organizational
Research Methods, 11, 326-352.
t
Stone-Romero, E. F. (1994). Construct validity issues in Stene-Romero, E. F., & Stone, D. L. (2007). Cognitive,
organizational behavior research. In]. Greenberg affective, and cultural influences on stigmatization
(Ed.), Organizational behavior: The state of the science and its impact on human resource management
(pp. 155-179). Hillsdale, NJ: Erlbaum. processes and practices. Research in Personnel and
Human Resource Management, 26,117-167.
Storie-Romero, E. F. (2002). The relative validity and use-
fulness of various empirical research designs. In S. G. Stone-Remere, E. F., Stone, D. L., & Hyatt, D. (2003).
Rogelberg (Ed.), Handbook of research methods in Personnel selection procedures and invasion of
industrial and organizational psychology (pp. 77-98). privacy. Journal of Social Issues, 59, 343-368.
Malden, MA: Blackwell. Stene-Romero, E. F., Weaver, A. E., & Glenar,]. L.
Storie-Romero, E. F. (2007a). Non-experimental designs. (1995). Trends in research design and data analytic
In S. Rogelberg (Ed.), The encyclopedia of industrial strategies in organizational research. Journal of
and organizational psychology (pp. 519-521). Beverly Management, 21, 141-157.
Hills, CA: Sage. Thomas, K. W., & Tymon, W. G. (1982). Necessary
Storie-Romero, E. F'. (2007b). Quasi-experimental properties of relevant research: Lessons from criti-
designs. In S. Rogelberg (Ed.), The encyclopedia cisms of the organizational sciences. Academy of
of industrial and organizational psychology Management Review, 7, 345-352.
(pp. 655-657). Beverly Hills, CA: Sage. Vandenberg, R. J., & Lance, C. E. (1992). Examining the
Storie-Romero, E. F. (2007c). Experimental designs. In causal order of job satisfaction and organizational
S. Rogelberg (Ed.), The encyclopedia of industrial and commitment. Journal of Management, 18, 153-167.
organizational psychology (pp, 239-241) Beverly Webb, E.]., Campbell, D. T., Schwartz, R. D., Sechrest,
Hills, C1: Sage. L., & Grove,]. B. (1981). Nonreactive measures in the
Stone- Romero, E. F. (2009). Implications of research social sciences (2nd ed.). Boston: Houghton Mifflin.
design options for the validity of inferences derived Weber, S.J., & Cook, 1. D. (1972). Subject effects in
from organizational research. In D. Buchanan & laboratory research: An examination of subject
A. Bryman (Eds.), Handbook of organizational roles, demand characteristics, and valid 'inference,
research methods (pp. 302-327). London: Sage. Psychological Bulletin, 77, 273-295.
Stone-Romero, E. F., & Anderson, L. (1994). Relative Weick, K. E. (1965). Laboratory experimentation with
power of moderated multiple regression and the organizations. In]. G. March (Ed.), Handbook of orga-
comparison of subgroup correlation coefficients for nizations (pp. 194-260). Chicago: Rand McNally.
detecting moderating effects. Journal of Applied Wiener, Y., & Vardi, Y. (1980). Relationships between
Psychology, 79, 354-359. job, organization, and career commitments and work
)

Stene-Romero, E. F., & Gallaher, L. (2006, May). outcomes: An integrative approach. Organizational
Inappropriate use of causal language in reports of non- Behavior and Human Performance, 26, 81-96.
experi~ental research. Paper presented at the meeting Williams, L. J., & Hazer,]' T. (1986). Antecedents and
of the Society for Industrial and Organizational consequences of satisfaction and commitment in
Psychology, Dallas, TX. turnover models: A reanalysis using latent variable
Stone-Romero, E. F., & Liakhovitski, D. (2002). Strategies structural equation methods. Journal of Applied
for detecting moderator variables: A review of theory Psychology, 71, 219-231.
and research. Research in Personnel and Human Yin, R. K. (1989). Case study research: Design and method.
Resources Management, 21, 333-372. Newbury Park, CA: Sage.
Storie-Romero, E. F., & Rosopa, P.]. (2004). Inference Zedeck, S. (1971). Problems with the use of "moderator"
problems with hierarchical multiple regression-based variables. Psychological Bulletin, 76, 295-310.
tests of mediating effects. Research in Personnel and
Zickar, M.J., & Slaughter,]. E. (2002). Computational
Human Resources Management, 23, 249-290.
modeling. In S. Rogelberg (Ed.), The encyclopedia of
Stene-Romero, E. F., & Rosopa,P.]. (2007). Tests of industrial and organizational psychology (pp, 184-197Jr
mediating effects. In N.]. Salkind (Ed.), The encyclo- Beverly Hills, CA: Sage. ..

72

: !
ZZEZ 55W