Assessment

http://asm.sagepub.com/

The NEO Personality Inventory−Revised: Factor Structure and Gender Invariance From Exploratory
Structural Equation Modeling Analyses in a High-Stakes Setting
Adrian Furnham, Nigel Guenole, Stephen Z. Levine and Tomas Chamorro-Premuzic
Assessment 2013 20: 14 originally published online 26 July 2012
DOI: 10.1177/1073191112448213

The online version of this article can be found at:

http://asm.sagepub.com/content/20/1/14

Published by:

http://www.sagepublications.com

Additional services and information for Assessment can be found at:

Email Alerts: http://asm.sagepub.com/cgi/alerts

Subscriptions: http://asm.sagepub.com/subscriptions

Reprints: http://www.sagepub.com/journalsReprints.nav

Permissions: http://www.sagepub.com/journalsPermissions.nav

Citations: http://asm.sagepub.com/content/20/1/14.refs.html

>> Version of Record - Jan 20, 2013

OnlineFirst Version of Record - Jul 31, 2012

OnlineFirst Version of Record - Jul 26, 2012

What is This?

Downloaded from asm.sagepub.com at EASTERN KENTUCKY UNIV on November 27, 2014

213
l.
ASM20110.1177/107319

Assessment

The NEO Personality Inventory–Revised:

20(1) 14­–23
© The Author(s) 2013
Reprints and permission:

Factor Structure and Gender Invariance sagepub.com/journalsPermissions.nav

DOI: 10.1177/1073191112448213
http://asm.sagepub.com
From Exploratory Structural Equation
Modeling Analyses in a High-Stakes Setting

Adrian Furnham1, Nigel Guenole2,

Stephen Z. Levine3, and Tomas Chamorro-Premuzic2

Abstract
This study presents new analyses of NEO Personality Inventory–Revised (NEO-PI-R) responses collected from a large
British sample in a high-stakes setting. The authors show the appropriateness of the five-factor model underpinning
these responses in a variety of new ways. Using the recently developed exploratory structural equation modeling (ESEM)
technique, the authors show that model fits improve markedly over conventional confirmatory factor analyses (CFA) of
the same data set, but that (a) factor interpretations do not change under ESEM analyses, (b) ESEM factor scores, just like
CFA factors scores, correlate at near unity with sums of observed scores, (c) NEO-PI-R facets under ESEM analyses are
invariant across gender, and (d) ESEM highlights the inappropriateness of alpha and beta as a higher order representation of
NEO-PI-R facets, whereas a CFA approach might lead researchers to believe in the appropriateness of these higher order
factors. These results, coupled with the existing validity evidence for the NEO-PI-R, suggest that the five-factor structure
is the most parsimonious structure for summarizing NEO-PI-R responses from high-stakes settings in the United Kingdom.

Keywords
NEO Personality Inventory–Revised, NEO-PI-R, Big Five, norms, personality traits, psychometrics

More than half a century since its discovery by early per- FFM such as honesty-humility (Ashton & Lee, 2008) and
sonality researchers, the five-factor model of personality ambition (Hogan & Chamorro-Premuzic, in press), the
(FFM) is now cemented as the dominant framework for facet-level structure of the FFM (Perugini & Gallucci,
describing consistent differences and similarities between 1997), and whether or not a neuropsychological basis exists
the ways people think, feel, and behave (Chamorro- for what is essentially a taxonomy of phenotypes (De
Premuzic, 2007; Goldberg, 1990, 1993; McCrae & Costa, Young, 2010).
1987). The literature underpinning the FFM is overwhelm- The NEO Personality Inventory–Revised (NEO-PI-R),
ing and the breadth of consensus on the value of the FFM is arguably the leading psychometric measure of the FFM, has
considerable (Chamorro-Premuzic & Furnham, 2010). This played a towering role in bringing researchers to this point
latter point is clearly illustrated by opening paragraphs in in our understanding of personality. Although the research
numerous journal articles, which contain strikingly similar base for the NEO-PI-R is vast, and findings have been con-
descriptions of how the FFM model is observed across cul- sistently replicated in numerous and diverse contexts, the
tures and languages as well as demographic variables such NEO-PI-R model has an Achilles’ heel: namely, that confir-
as gender and age (e.g., Saucier & Goldberg, 1998; Saucier matory factor analyses (CFAs) do not yield evidence in
& Ostendorf, 1999). In fact, apart from a few notable excep-
tions (e.g., Block, 1995, 2001), disputes over the FFM’s 1
University College London, London, UK
legitimacy today are centered on issues of refinement rather 2
Goldsmiths, University of London, London, UK
3
than questioning the fundamental utility of the model. Some Bar Ilan University, Ramat Gan, Israel
of the aspects that are still being debated relate to the exis-
Corresponding Author:
tence or otherwise of a higher order structure of the FFM Adrian Furnham, University College London, 26 Bedford Way, London,
(e.g., Digman, 1997; Rushton & Irwing, 2009), the status of WC1H 0AP, UK
personality dimensions that are not well represented by the Email: [email protected]

Downloaded from asm.sagepub.com at EASTERN KENTUCKY UNIV on November 27, 2014

Furnham et al. 15

Figure 1. Confirmatory factor analysis (CFA) versus exploratory structural equation modeling (ESEM) representations of a two-factor
model

support of the FFM when judged by traditionally accepted assumes zero loadings on the nontarget factor. On the other
psychometric standards (Church & Burke, 1994; McCrae, hand, the ESEM model in panel 1B allows nonzero load-
Zonderman, Costa, Bond, & Paunonen, 1996). In fact, the ings on factors other than the primary targeted factor, as
fit of CFA models to NEO-PI-R data has routinely been so illustrated by the dotted arrows. It is thought that this flexi-
poor that McCrae et al. (1996) suggested abandoning CFA. bility is more realistic, and these relaxed assumptions are
The inability of researchers to fit an appropriate CFA model expected to lead to improved fit.
means that there still remain researchers who argue for both The need for a methodological innovation such as ESEM
fewer (Rushton & Irwing, 2009) and more (Ashton & Lee, to reconcile the conflicting evidence from CFA analyses of
2008) factors of personality. This is in part because of the NEO-PI-R data sets and other methods of validation is well
recognition that inappropriate use of CFA may result in illustrated by research carried out on the short form of the
overestimation or underestimation of the number of person- NEO-PI-R. The shorter NEO-FFI (NEO Five-Factor
ality factors extracted. Inventory; 60 vs. 240 items) attracts considerable research
Exploratory structural equation modeling (ESEM), a interest, no doubt because of its quicker administration time
recent development in psychological measurement, offers (less than 10 minutes vs. more than 35 minutes for the lon-
the potential to reconcile these seemingly opposing vantage ger NEO-PI-R). Egan, Deary, and Austin (2000), however,
points. This is because ESEM has a number of advantages in addition to providing British norms based on 1,025
over traditional CFA approaches that mean it could be more adults, suggested that the instrument “requires modification
appropriate for modeling personality data. These advan- and improvement before it can be regarded as measuring
tages include relaxation of the assumption that items have the five independent personality traits” (p. 907). In a recent
factorial complexity of one (i.e., no cross-loadings of items article, Marsh et al. (2010) used the ESEM technique, cou-
or facets), the availability of standard errors for parameter pled with theoretically appropriate correlated residuals, to
estimates in an exploratory setting, and an assessment of fit examine fit for the NEO-FFI. Results showed that using
using goodness-of-fit indices available in traditional struc- ESEM on the NEO-FFI provided a much better model fit
tural equation modeling frameworks (Asparouhov & than that CFA has exhibited up until this point.
Muthén, 2009; Marsh et al., 2010). The flexibility of ESEM, Despite the fact that ESEM has been shown to result in
as contrasted with CFA, is illustrated in Figure 1 for a hypo- improved model fit for the FFM over traditional CFA
thetical two-factor model. The CFA model in panel 1A approaches, several questions of fundamental importance

Downloaded from asm.sagepub.com at EASTERN KENTUCKY UNIV on November 27, 2014

16 Assessment 20(1)

to assessment specialists using the NEO-PI-R remain purpose of personnel selection or career progression (inter-
unanswered. First, no studies exist that apply ESEM to lon- nal promotion), representing a high-stakes testing scenario
ger forms of the NEO-PI-R. As a consequence, there is no as results were used to inform these consequential deci-
evidence as to whether the improved fit observed for the sions. Participants’ age ranged from 18 to 67 years and
NEO-FFI also exists for the NEO-PI-R. There is also no most participants were employed as middle or senior man-
evidence on whether the FFM primary factor interpreta- agers in a range of British-based companies. Of the 13,234
tions change as a result of the application of ESEM. adults who participated in the assessment center, 4,937
Critically, a further issue that remains to be investigated is participants completed the NEO-PI-R and were included in
whether ESEM produces person scores that are well the current study, out of whom 25% were female. The
approximated by the simple sums of candidates’ observed average age of the sample was 44 years and the standard
scores. Because these simple sums are the most routinely deviation was 14 years.
interpreted scores in applied settings, the answer to this
question has important implications for the accuracy of
norm data that is so integral to attributing meaning to per- Measure
sonality profiles. Although Marsh, Liem, Martin, Morin, The NEO-PI-R (Costa & McCrae, 1992) questionnaire is a
and Nagengast (2011) examined gender invariance for the 240-item questionnaire designed to measure the FFM traits
NEO-FFI, no study has yet examined whether factor scores as well as six primary facets for every trait. Items are
from ESEM models, with their improved fits, yield FFM responded on a 5-point Likert-type scale that ranged from
scores on the long form NEO-PI-R that are invariant across strongly disagree to strongly agree. The test is untimed but
gender. Finally, we expect researchers and practitioners takes approximately 35 minutes to complete. Although a
alike would like to know if an alternative higher order wealth of research exists providing evidence for the validity
structure exists for more parsimoniously describing per- and the reliability of this instrument, most data are derived
sonality based on FFM dimensions derived using the more from student and low-stakes settings (Chamorro-Premuzic
flexible ESEM approach. Several higher order structures of & Furnham, 2010).
the FFM have been proposed. In a seminal article, Digman
(1997) proposed that covariation between the higher order
factors of Extraversion and Openness could be explained Procedure
by a factor he labeled alpha, while covariation among the Participants were tested in an assessment center setting for
remaining facets of Neuroticism, Agreeableness, and selection and promotion purposes. The questionnaire was
Conscientiousness could be explained by a factor he called untimed and most participants took between 30 and
beta. DeYoung (2010) labeled similar higher order struc- 45 minutes to complete it. They were asked to respond
tures plasticity and stability based on psychometric as well honestly and were promised, and received, full feedback on
as neuropsychological evidence. More recently, propo- their scores at a later point.
nents of a general factor of personality have presented psy-
chometric evidence for what must be the most basic
representation of personality so far (Rushton & Irwing, Analyses
2009). Given the increased flexibility of ESEM over tradi- Descriptive analyses. SPSS 17.0 was used for data clean-
tional CFA methods, it would be of interest to practitioners ing. The calculation of the correlations and the simple
to know whether there is a more parsimonious and well- descriptive analyses are presented in tables later in the text.
fitting representation of the FFM that they might use in Structural equation models. All structural equation mod-
their applied work based on the factor scores emerging eling was carried out using the Mplus computer program
from ESEM. In this article, we answer these questions (Version 6.1; Muthén & Muthén, 2006) with the MLR
using a large sample of white-collar British workers who estimator to handle issues related to nonnormality of the
completed the NEO-PI-R in a high-stakes selection setting. data. We first fitted both CFA and ESEM models to males
and females separately, prior to examining measurement
equivalence, which must be demonstrated before conclud-
Method ing that survey items measure constructs similarly across
Participants populations. Geomin rotation, an oblique rotation method
allowing the factors to be intercorrelated, was performed
In all, 13,234 British adults were tested over a 10-year in the case of ESEM models. The consensus today among
period as part of an assessment center run by chartered methodologists is that oblique rotation should always be
organizational psychologists. Each participant completed a preferred. This is because we most constructs in social sci-
number of self-report and ability tests as well as other ences turn out to be intercorrelated, and an oblique rota-
exercises and an interview. Tests were completed for the tion will uncover an orthogonal (i.e., uncorrelated)

Downloaded from asm.sagepub.com at EASTERN KENTUCKY UNIV on November 27, 2014

Furnham et al. 17

solution if one is appropriate anyway (MacCallum, 1998). Table 1. Fit Statistics for Traditional Confirmatory Factor
CFA does not use rotations because the patterns of fixed Analysis Models
and free loadings are specified a priori.
CFA χ2 df p RMSEA CFI TLI SRMR
Measurement equivalence. Measurement equivalence pre-
vails if two individuals with equal standing on the construct Male 17640.528 395 <.01 .11 .65 .60 .12
assessed, but sampled from different populations (i.e., gen- Female 5920.116 395 <.01 .11 .65 .61 .12
ders), have equal expected observed scores on the mea- Baseline 24293.692 815 <.01 .11 .63 .60 .12
surement instrument (Drasgow, 1984). Today, numerous Note. df = degrees of freedom; RMSEA = root mean square error of
publications exist that outline the same steps for showing approximation; CFI = comparative fit index; TLI = Tucker–Lewis index,
measurement invariance in multiple group models (e.g., SRMR = standardized root mean square residual.
Horn & McArdle, 1992; Millsap, 1995; Vandenberg &
Lance, 2000). First, an unconstrained baseline model is esti-
mated where the only parameters equated across genders Table 2. Fit Statistics for Exploratory Structural Equation
Models
are those required for identification purposes. If a satisfac-
tory fit for the baseline model is observed, configural, or Model χ2 df p RMSEA CFI TLI SRMR
weak invariance, is said to hold. In other words, the same
Male 4760.87 295 <.01 .06 .91 .86 .03
number of factors exists in the data from both groups and Female 1634.01 295 <.01 .06 .91 .87 .03
items have the same pattern of zero and nonzero loadings in Baseline 6394.87 590 <.01 .06 .91 .86 .03
both groups. Next, factor loadings are constrained to be Metric 6711.34 715 <.01 .06 .90 .88 .04
equal across both groups. If constraints on the factor load- Scalar 7205.04 740 <.01 .06 .90 .88 .04
ings do not reduce fit appreciably, metric, or strong invari-
ance is said to hold. Next, the intercepts for the items are Note. df = degrees of freedom; RMSEA = root mean square of
approximation; CFI = comparative fit index; TLI =Tucker–Lewis index,
held equal across groups. If the item intercepts constraints SRMR =standardized root mean square residual.
do not appreciably reduce fit, strict, or scalar invariance is
said to hold. Although further constraints related to the fac-
tor variances, factor covariances, and residuals are possible models for males and females fitted the data poorly when
(cf. Marsh et al., 2011), fulfillment of these additional con- compared with the conventional standards outlined above.
straints are not requirements necessary for comparisons of This pattern of poor fit was also observed for the baseline
mean differences on the construct under study. model for configural invariance. In particular, in addition to
Model selection. We report multiple indices in addition to chi-square statistics, which are highly significant in all
the model chi-square, because its sensitivity to sample size cases, RMSEA is more than .10, which indicates poor
can lead to rejection of theoretically appropriate models model fit. Moreover, CFI and TLI are clearly not anywhere
(e.g., Byrne, 1998). We selected the most appropriate model near .90 and .95. Because of this, we did not proceed fur-
out of these sequences of models based on an overall assess- ther to investigate metric and scalar invariance, as CFA
ment of the following indices: root mean square error of models are inappropriate for these data.
approximation (RMSEA; Steiger, 1990), Bentler’s (1990)
comparative fit index (CFI), and Bentler and Bonnet’s
(1980) nonnormed fit index (NNFI). The RMSEA is a mea- Exploratory Structural
sure of badness of fit per degree of freedom, and conven- Equation Modeling Results
tionally values less than .05 are considered indicative of Results from ESEM analyses indicated an improvement in
good fit (Brown & Cudeck, 1993). The CFI and Tucker– fit for the male-only and female-only models to what can
Lewis index (TLI) range between 0 and 1 and values more be argued to be acceptable levels by conventionally
than .9 are considered acceptable fit, whereas values more accepted standards. The improvement in fit between the
than .95 indicate excellent fit (Hu & Bentler, 1995). Impor- CFA and ESEM models is approximately comparable in
tantly, Marsh, Hau, and Wen (2004) have suggested that magnitude with the changes that Marsh et al. (2010)
these standards are unlikely to be achieved with CFA when observed by moving from CFA to ESEM on the short
models as complex as the FFM are analyzed. form of the NEO-PI-R, the NEO-FFI. The better model fit
observed here is most likely because of the relaxation of
CFA conditions where each facet is only allowed to load
Results on its target factor and has zero loadings on every other
Confirmatory Factor Analysis Results factor. Marsh et al. (2010) referred to this CFA model as
the independent clusters CFA model. Table 2 shows that
Table 1 presents the results of CFA analyses of facet-level for the separate male and female ESEM models, while
data using Mplus. These results show that the separate chi-square remains highly significant; RMSEA and CFI

Downloaded from asm.sagepub.com at EASTERN KENTUCKY UNIV on November 27, 2014

18 Assessment 20(1)

now indicate adequate fit, whereas TLI is very close to the detract from model fit. In addition to N5 not having its high-
acceptable level. est loading on Factor I, which is ostensibly Neuroticism, the
facet N6 vulnerability has nontrivial loadings on Factor V
for men and women. Using the same criteria, Factor II is
Measurement Equivalence Results Extraversion. All Extraversion facets except E3 assertive-
The model fit for the baseline model testing configural ness have their strongest loading on Factor II, and all load-
invariance with ESEM, presented in Table 2, indicates fit ings for Extraversion facets on Factor II are significant for
statistics similar to the single-group ESEM results. Because men and women. E3 not only has a greater loading in both
this is a substantial improvement over CFA results, and fit the male and female samples on Factor IV than it does on
statistics approach accepted standards, we proceeded to Factor II but it also has a smaller and still considerable load-
examine metric and scalar invariance. The results in Table 2 ing on Factor V.
show that the imposition of metric and scalar invariance The preponderance of evidence points compellingly
constraints does not reduce fit appreciably. These findings toward Factor III representing the FFM Openness dimen-
show that the ESEM solution is a more appropriate solution sion. For males and females, all Openness facets have their
than the CFA solution for these data, and importantly, that strongest loading on Factor III except O3 feelings, which
there is no differential facet functioning in this data set loads higher on Factor II (that we have labeled Extraversion)
across gender. The correlations between the factors for the in both men and women. Moreover, the need for flexibility
scalar invariance ESEM solution were small to moderate. in modeling nontarget loadings is again illustrated by siz-
The largest correlation, at −.45, was between Factor I and able secondary loadings for O6 values. Factor IV in the
Factor V. The smallest correlation, at −.06, was between ESEM solution is Agreeableness, because the highest load-
Factor IV and Factor V. ing for each agreeableness facet is on Factor IV, and all
these loadings are significant. Here again, sizable second-
ary loadings show the inappropriateness of the independent
Interpretation of Exploratory clusters model for these data. Factor V emerges strongly as
Structural Equation Modeling Factors Conscientiousness with all conscientiousness facets having
One of the primary advantages of the ESEM approach is their highest loadings on the final factor, Factor V, all of
that it relaxes the assumption that items or facets have zero which are significant. In sum, the geomin rotated ESEM
loadings on all factors other than the target factor. This solutions for men and women reveal two important points.
model restriction is unrealistic for personality data, which First, ESEM reveals clear support for the FFM. Second,
are known to be factorially complex (Marsh et al., 2007). substantial cross-loading for certain facets, most notably
Along with the increased flexibility, however, comes the N5, E3, and O3, shows the inappropriateness of the inde-
need to interpret the factors as one would in an exploratory pendent clusters model assumptions of traditional CFA
factor analyses. In other words, it is quite possible under approaches if the goal is to obtain strong fit from structural
ESEM that the pattern of factor loadings will not support equation modeling.
the a priori FFM structure and patterns of factor loadings
need to be examined. Interpretation of the factors that
emerge from ESEM ultimately requires judicious interpre- Is There a Higher Order Structure
tation of the loading pattern and significance of the load- Based on the NEO-PI-R’s ESEM Solution?
ings for each of the facets. It seems reasonable, however, to Although second-order ESEM factor models are not cur-
have as our requirement that for a factor to be considered rently possible in Mplus, it is possible to save the factor
an a priori component of the FFM, all, or at least the major- scores from ESEM and subject these to further analysis
ity of the facets that measure the factor ought to have their using ESEM. We used this procedure to investigate one-
highest loadings on it, and that all these loadings should be and two-factor representations of the factor scores that
significant. emerged from the best fitting CFA and ESEM models
By these criteria, we see that Factor I is Neuroticism. For described earlier. First, the single-factor model based on
both males and females, all the loadings of the N facets are CFA scores did not converge, suggesting that a single-
significant on Factor I, and moreover, all facets except N5 factor model is inappropriate for these data. The single-factor
Impulsiveness have their highest loadings on Factor I (Table 3). model for the ESEM scores did converge, but fit was so bad
For males, N5 has a higher loading on Factor II and Factor V as to preclude further interpretation (RMSEA = .32, CFI = .71,
than it does on Factor I, and only a marginally smaller load- TLI = .43). The two-factor model based on CFA factor
ing on Factor IV. For females, N5 has a greater loading on scores did converge. The factors clearly resembled alpha or
Factor V and substantial loadings on Factor II and Factor plasticity with high Extraversion and Openness loadings
IV. Constraining these loadings to zero under the indepen- (N = −.01, E = .77, O = .87, A = .21), and beta or stabil-
dent clusters model that underpins CFA will certainly ity with high-reverse Neuroticism, Agreeableness, and

Downloaded from asm.sagepub.com at EASTERN KENTUCKY UNIV on November 27, 2014

Furnham et al. 19

Table 3. Loading Parameter Estimates and Significance From Standardized ESEM Solution
Male Female
Facet Est. p Est. p Est. p Est. p Est. p Est. p Est. p Est. p Est. p Est. p
N1 .79 .00 .01 .54 .08 .00 .03 .05 .00 .96 .82 .00 .01 .54 .07 .00 .03 .05 .00 .96
N2 .65 .00 .02 .39 .01 .47 −.48 .00 −.03 .24 .71 .00 .01 .40 .01 .47 −.46 .00 −.03 .24
N3 .73 .00 −.05 .06 .10 .00 .00 .75 −.18 .00 .74 .00 −.04 .07 .09 .00 .00 .75 −.16 .00
N4 .61 .00 −.17 .00 .01 .53 .09 .00 −.06 .03 .62 .00 −.14 .00 .01 .53 .08 .00 −.06 .03
N5 .34 .01 .40 .00 −.01 .49 −.25 .00 −.41 .00 .37 .01 .36 .00 .00 .49 −.24 .00 −.39 .00
N6 .54 .00 −.03 .39 −.05 .01 .11 .00 −.36 .00 .56 .00 −.02 .39 −.05 .01 .10 .00 −.33 .00
E1 .00 .94 .84 .00 .00 .76 .35 .00 .09 .06 .00 .94 .84 .00 .00 .76 .36 .00 .09 .06
E2 −.05 .48 .68 .00 −.13 .00 .06 .03 .02 .62 −.05 .48 .66 .00 −.13 .00 .06 .03 .02 .62
E3 −.15 .00 .32 .00 .00 .68 −.42 .00 .27 .00 −.17 .00 .29 .00 .00 .68 −.40 .00 .26 .00
E4 −.01 .78 .33 .00 .09 .00 −.31 .00 .32 .00 −.01 .78 .31 .00 .09 .00 −.30 .00 .32 .00
E5 −.05 .40 .38 .00 .10 .00 −.20 .00 −.10 .00 −.05 .40 .33 .00 .10 .00 −.18 .00 −.10 .00
E6 −.04 .34 .69 .00 .13 .00 .09 .00 −.02 .59 −.05 .34 .70 .00 .14 .00 .09 .00 −.02 .59
O1 .01 .79 .19 .00 .52 .00 −.06 .00 −.37 .00 .01 .79 .18 .00 .53 .00 −.06 .00 −.37 .00
O2 −.01 .87 .00 .48 .67 .00 .13 .00 −.06 .01 −.01 .87 .00 .49 .71 .00 .13 .00 −.06 .01
O3 .31 .00 .51 .00 .39 .00 −.01 .56 .03 .43 .36 .00 .51 .00 .41 .00 −.01 .55 .03 .43
O4 −.26 .00 .24 .00 .39 .00 .00 .91 −.12 .00 −.30 .00 .23 .00 .41 .00 .00 .91 −.13 .00
O5 −.21 .02 −.14 .00 .70 .00 .01 .49 .01 .39 −.23 .02 −.13 .00 .70 .00 .01 .50 .01 .40
O6 −.20 .00 .16 .00 .27 .00 .01 .47 −.16 .00 −.24 .00 .17 .00 .29 .00 .01 .48 −.17 .00
A1 −.24 .00 .34 .00 .03 .11 .42 .00 −.02 .24 −.26 .00 .31 .00 .03 .11 .40 .00 −.02 .24
A2 −.01 .87 .04 .13 −.05 .00 .55 .00 .11 .00 −.01 .87 .04 .13 −.05 .00 .53 .00 .11 .00
A3 .06 .29 .55 .00 .00 .81 .62 .00 .24 .00 .07 .29 .52 .00 .00 .81 .61 .00 .25 .00
A4 −.20 .00 .03 .42 .02 .30 .73 .00 −.01 .53 −.23 .00 .03 .42 .02 .30 .72 .00 −.01 .54
A5 .09 .01 −.01 .81 −.03 .10 .47 .00 −.04 .09 .10 .01 −.01 .81 −.03 .10 .47 .00 −.04 .09
A6 .09 .03 .22 .00 .14 .00 .53 .00 .01 .47 .11 .03 .21 .00 .15 .00 .53 .00 .01 .47
C1 −.16 .23 .03 .33 .14 .00 −.02 .26 .64 .00 −.18 .23 .02 .34 .13 .00 −.01 .26 .62 .00
C2 .21 .11 −.08 .00 −.02 .29 −.01 .41 .67 .00 .23 .11 −.07 .00 −.02 .29 −.01 .41 .65 .00
C3 .04 .79 −.01 .57 −.05 .01 .14 .00 .76 .00 .05 .79 −.01 .57 −.05 .01 .14 .00 .74 .00
C4 .04 .75 .07 .05 .12 .00 −.24 .00 .68 .00 .04 .75 .07 .05 .12 .00 −.23 .00 .67 .00
C5 −.09 .55 −.01 .62 .02 .24 .00 .99 .77 .00 −.10 .55 −.01 .62 .02 .24 .00 .99 .73 .00
C6 −.05 .78 −.35 .00 .00 .80 .24 .00 .62 .00 −.05 .78 −.31 .00 .00 .80 .22 .00 .58 .00
Note. Est. = estimated geomin rotated factor loading; p = two-tailed p value. N = Neuroticism; N1 = Anxiety; N2 = Angry Hostility; N3 = Depression;
N4 = Self-Consciousness; N5 = Impulsiveness; N6 = Vulnerability; E = Extraversion; E1 = Warmth; E2 = Gregariousness; E3 = Assertiveness; E4 =
Activity; E5 = Excitement Seeking; E6 = Positive Emotion; O = Openness to Experience; O1 = Fantasy; O2 = Aesthetics; O3 = Feelings; O4 = Actions;
O5 = Ideas; O6 = Values; A = Agreeableness; A1 = Trust; A2 = Straightforwardness; A3 = Altruism; A4 = Compliance; A5 = Modesty; A6 = Tender
Mindedness; C = Conscientiousness; C1 = Competence; C2 = Order; C3 = Dutifulness; C4 = Achievement Striving; C5 = Self-Discipline; C6 =
Deliberation.

Conscientiousness loadings (N = −.83, E = .45, O = −.01, the first-order ESEM factors would not converge because
A = .18, and C = .80). Although we might conclude the of nonpositive definite covariance matrix, suggesting that
existence of higher order factors on the basis and fit the higher order solution was not appropriate. Thus, the
(RMSEA = .13, CFI = .99, TLI = .91), something is clearly application of ESEM highlights the inappropriateness of a
wrong at a chi-square of 88.86 and a single degree of free- higher order solution for the Big Five based on these data.
dom. Moreover, these values inspected casually would The rationale we offer for this result is an intriguing expla-
disguise the fact that the values are based on an even worse nation suggested by Ashton, Lee, Goldberg, and De Vries
fitting first-order model (i.e., the multiple-group CFA base- (2009). The essence of the argument of Ashton et al. is that
line model presented in Table 1). Despite evidence of alpha suppressed secondary loadings on nontarget constructs can
and beta based on loading patterns, the model fit and lead to correlations between factors. Although these corre-
knowledge that the factor solution is based on a poorly fit- lations among factors can be modeled by higher order
ting first-order solution demands caution before interpret- factors, the higher order factors accounting for these cor-
ing the supposed alpha and beta factors as substantive factors. relations will be spurious, because the correlations on
Importantly, the two-factor ESEM solution that analyzed which they are based are artifactual. On the basis of these

Downloaded from asm.sagepub.com at EASTERN KENTUCKY UNIV on November 27, 2014

20 Assessment 20(1)

Table 4. Correlations Among ESEM, CFA, and Observed Agreeableness factor, where the correlations are still in
Variable Scores excess of .90 for the CFA-observed correlations, but the
correlations between the ESEM and CFA and ESEM and
Domain CFA–OBS ESEM–OBS CFA–ESEM
observed variable scores drop to .80 and .84, respectively.
Neuroticism .97 .96 .97 However, we expect that even these correlations of .80 and
Extraversion .94 .93 .95 above will give many practitioners the confidence to
Openness .96 .97 .92 continue using observed variable scores for even the
Agreeableness .96 .84 .80 Agreeableness dimension of the NEO-PI-R.
Conscientiousness .96 .97 .98
Note. CFA–OBS = Correlation between confirmatory factor analysis
factor scores and corresponding observed variable sums; ESEM–OBS =
Demographic Descriptive
correlation between exploratory structural equation modeling factor for the Current British Sample
scores and corresponding observed variable sums; CFA–ESEM =
correlations between confirmatory factor analysis factor scores and Applied measurement specialists reading this article might
exploratory structural equation modeling factor scores. be interested in descriptive statistics for this sample to
enable them to calculate norms for interpretation in their
own work. They might also be interested to examine gender
differences or age-related patterns of association with per-
results, and if well-fitting models are of concern to practi- sonality. In response, in Table 5, we present means for the
tioners, the most general level of the FFM that yields ade- overall sample as well as for males and females separately.
quate and defensible fit, so far, is at the level of the five Effect sizes, calculated as Cohen’s d, are presented for
factors. gender. These show that overall the effect sizes are small to
moderate, with the largest effect size at the factor level
occurring for the Openness domain (d = .52 favoring
Correlations Between ESEM Factors, CFA females) whereas the largest facet-level effect size occurred
Factors, and Observed Variable Totals on O3 feelings (d = .55 favoring females). The magnitude
The way that norms for the NEO-PI-R are typically used, of the differences observed, where just a handful of facets
as is the case with other personality tests, involves creating show moderate-sized differences of a half standard devia-
observed item totals for candidates and calculating the pro- tion, mirrors the findings of Costa, Terracciano, and
portion of a representative norm sample that scores the McCrae (2001). Costa et al. reported that the largest gender
same as or lower than the candidate. It is today well known difference for the NEO-PI-R was .44 for the N6 vulnerabil-
that correlations between latent variable scores and ity facet of Neuroticism. Other similarities also exist
observed variable totals are often very high (Fan, 1998). between our study and theirs, for example, females are
Correlations between latent variable scores and external uniformly higher on all Neuroticism facets in both studies.
criteria and observed variable scores and external criteria The United Kingdom, however, was not represented in
are also known to be similar (Ferrando & Chico, 2007), Costa et al.’s study. We also recommend caution in making
justifying use of observed variable scores in the scoring and comparisons as it is today clear that the purpose of the per-
feedback process. However, assessment practitioners who sonality testing has an impact on the mean levels observed
use the NEO-PI-R will want to know whether the observed for personality measures (e.g., De Fruyt et al., 2006). That
variable scores they routinely use are still appropriate, now is to say, score inflation in selection settings is likely to
that a more flexible latent variable modeling approach has render comparisons with data sets from lower stakes per-
yielded better model fit to the NEO-PI-R. sonality testing situations inappropriate.
To investigate this issue, we saved the factor scores from Table 5 contains the correlation between the NEO-PI-R
both the poor fitting CFA models along with the better fit- domain scores, along with its facets, and age. These results
ting ESEM models, and correlated each set of scores both show that, by and large, there are only weak associations
with each other and with their observed variable equiva- between personality and age. Where significant correlations
lents. Results, presented in Table 4, indicated that the cor- between age and personality are observed, for example with
relations between both forms of latent scores (i.e., ESEM agreeableness, the results show that the correlations are in
factor scores or CFA factor scores) and observed variable fact quite small. This means that there is only a very minor
scores were near unity in almost all instances. This suggests association between personality and age. Moreover, signifi-
that practitioners may continue to use observed scores in cant associations when the correlations are so small suggest
their work because the improved model fit for the ESEM that they are the result of high power from the large sample
scores does not substantially affect the scores that emerge size, rather than showing a substantively meaningful rela-
from these analyses. The exception to this pattern is for the tionship between age and personality.

Downloaded from asm.sagepub.com at EASTERN KENTUCKY UNIV on November 27, 2014

Furnham et al. 21

Table 5. Sample Descriptive Statistics for NEO-PI-R Facets

Scale Overall Male (n = 3,715) Female (n = 1,222) Cohen’s d Effect Size Age
r Mean SD Mean SD Mean SD d 95% LB 95% UB r p
N 65.56 19.42 63.98 18.99 70.39 19.95 −.33 −.87 .20 .01 .79
N1 12.44 5.14 12.01 5.02 13.75 5.28 −.34 −.48 −.20 .01 .51
N2 10.17 4.53 10.02 4.49 10.63 4.60 −.13 −.26 −.01 .01 .57
N3 9.69 4.66 9.44 4.55 10.42 4.92 −.21 −.34 −.10 −.02 .36
N4 11.86 4.22 11.68 4.15 12.42 4.38 −.18 −.29 −.06 .00 .90
N5 14.75 4.36 14.44 4.34 15.68 4.30 −.28 −.41 −.16 .00 .86
N6 6.72 3.46 6.42 3.38 7.62 3.54 −.35 −.44 −.25 .02 .47

E 127.14 18.22 126.12 18.37 130.23 17.42 −.23 −.73 −.28 −.04 .06
E1 23.73 3.93 23.22 3.94 25.30 3.48 −.54 −.65 −.44 −.01 .62
E2 20.05 4.63 19.79 4.67 20.84 4.41 −.23 −.36 −.10 −.01 .52
E3 20.55 4.55 20.73 4.49 19.98 4.66 .17 .04 .29 −.01 .64
E4 21.54 4.12 21.44 4.15 21.85 3.99 −.10 −.22 −.01 −.02 .43
E5 18.77 4.49 18.89 4.44 18.42 4.63 −.11 −.02 .23 −.06 .00
E6 22.51 4.58 22.07 4.62 23.82 4.19 −.39 −.51 −.26 −.04 .07

O 120.96 18.52 118.63 18.24 128.05 17.54 −.52 −1.03 −.02 −.02 .49
O1 16.93 4.78 16.55 4.76 18.06 4.65 −.32 −.45 −.19 .00 .84
O2 17.76 5.95 17.11 5.96 19.75 5.46 −.45 −.62 −.29 .01 .58
O3 21.84 4.19 21.28 4.17 23.52 3.81 −.55 −.66 −.44 −.02 .37
O4 20.10 4.18 19.67 4.20 21.44 3.83 −.43 −.55 −.32 −.01 .77
O5 20.54 5.25 20.39 5.26 21.00 5.20 −.12 −.26 .03 −.01 .54
O6 23.79 3.42 23.65 3.48 24.22 3.17 −.17 −.26 −.07 −.03 .18

A 118.53 15.67 117.13 15.65 122.79 14.96 −.37 −.80 .07 −.05 .01
A1 21.74 4.06 21.63 4.04 22.07 4.11 −.11 −.22 .01 −.04 .09
A2 18.36 4.45 18.10 4.46 19.16 4.31 −.24 −.36 −.12 −.06 .01
A3 23.68 3.55 23.30 3.56 24.83 3.27 −.44 −.53 −.34 −.03 .11
A4 18.15 4.01 17.99 4.03 18.64 3.93 −.16 −.27 −.05 −.01 .65
A5 17.10 4.55 16.85 4.58 17.88 4.37 −.23 −.36 −.10 −.01 .58
A6 19.49 3.50 19.27 3.54 20.18 3.29 −.26 −.36 −.16 −.05 .03

C 132.97 17.38 133.32 17.41 131.91 17.24 −.08 −.40 −.57 .00 .84
C1 24.05 3.23 24.15 3.19 23.76 3.30 .12 .03 .21 .02 .40
C2 18.97 4.51 18.87 4.52 19.27 4.47 −.09 −.21 −.04 −.01 .63
C3 24.65 3.56 24.77 3.54 24.28 3.60 .14 .04 .24 −.02 .35
C4 22.97 4.02 23.01 4.04 22.84 3.98 .04 −.07 .15 −.03 .23
C5 23.81 4.05 23.78 4.04 23.93 4.09 −.04 −.15 .07 −.02 .35
C6 18.54 4.41 18.77 4.36 17.84 4.50 .21 .09 .33 .03 .12
Note. NEO-PI-R = NEO Personality Inventory–Revised; LB = lower bound; UB = upper bound; N = Neuroticism; N1 = Anxiety; N2 = Angry Hostility;
N3 = Depression; N4 = Self-Consciousness; N5 = Impulsiveness; N6 = Vulnerability; E = Extraversion; E1 = Warmth; E2 = Gregariousness; E3 =
Assertiveness; E4 = Activity; E5 = Excitement Seeking; E6 = Positive Emotion; O = Openness to Experience; O1 = Fantasy; O2 = Aesthetics; O3 =
Feelings; O4 = Actions; O5 = Ideas; O6 = Values; A = Agreeableness; A1 = Trust; A2 = Straightforwardness; A3 = Altruism; A4 = Compliance; A5 =
Modesty; A6 = Tender Mindedness; C = Conscientiousness; C1 = Competence; C2 = Order; C3 = Dutifulness; C4 = Achievement Striving; C5 = Self-
Discipline; C6 = Deliberation.

Discussion have not been able to show satisfactory model fit for it
using modern psychometric methods such as CFA. The
Although the NEO-PI-R is one of the most well researched current results highlight that CFA, the most common
instruments available for the assessment of broad dimensions method to analyze the FFM factors, makes unrealistic
of personality, and large norm databases exist, researchers assumptions with regard to the factorial complexity of each

Downloaded from asm.sagepub.com at EASTERN KENTUCKY UNIV on November 27, 2014

22 Assessment 20(1)

of the NEO-PI-R facets. These facets clearly have nonzero that, tests such as the NEO-PI-R are used so widely in occu-
loadings on numerous factors, and this violates what Marsh pational settings. Because the gender differences were
et al. (2007) referred to as the independent clusters model. small and associations with age were weak, the descriptive
Importantly, we showed that the factors retain their a priori statistics presented here might be used by industrial–
interpretations when modeled at facet level using ESEM. organizational psychologists wanting to create norms for
The factor scores estimated based on the well-fitting ESEM high-stakes selection in the United Kingdom without too
model were then shown to be correlated almost perfectly much concern over impact against women and older work-
with both the scores estimated based on the CFA model and ers at mid- to advanced career stages.
their observed variable total counterparts. The biggest dis-
crepancy occurred for agreeableness, although correlations Declaration of Conflicting Interests
were still large (.80 with CFA scores and .84 with ESEM The authors declared no potential conflicts of interest with respect to
scores). The ESEM model also showed measurement the research, authorship, and/or publication of this article.
equivalence for the number of factors (configural invari-
ance), facet loadings (metric invariance), and facet inter- Funding
cepts (scalar invariance). This suggests that the relation The authors received no financial support for the research, author-
between facet-level scores and the latent personality dimen- ship, and/or publication of this article.
sion is the same for both males and females. Finally, analy-
ses of the ESEM factors revealed a structure that resembled References
Digman’s (1997) alpha and beta for ESEM analyses of Ashton, M. C., & Lee, K. (2008). The prediction of honesty-
factor scores derived from CFA, but this model fitted humility-related criteria by the HEXACO and five-factor
poorly, and was based on factor scores from a first-order models of personality. Journal of Research in Personality,
model that had even worse fit. Under ESEM analyses of 42, 1216-1228.
first-order factor scores derived from a well-fitting first- Ashton, M. C., Lee, K., Goldberg, L. R., & de Vries, R. E. (2009).
order model, a two-factor model was shown to be inappro- Higher order factors of personality: Do they exist? Personality
priate because of inadmissible solutions. The most general and Social Psychology Review, 13, 79-91.
level of interpretation appropriate for responses to the Asparouhov, T., & Muthén, B. (2009). Exploratory structural equa-
NEO-PI-R, therefore, is still the FFM if model fit is of tion modeling. Structural Equation Modeling, 16, 397-438.
concern. Bentler, P. M. (1990). Comparative fit indexes in structural models.
Limitations of this study ought to be mentioned to facili- Psychological Bulletin, 107, 238-246.
tate interpretation of results and guide future research. First, Bentler, P. M., & Bonnet, D. C. (1980). Significance tests and
these data are collected in a setting where individuals were goodness of fit in the analysis of covariance structures. Psy-
being considered for selection and promotion. Results chological Bulletin, 88, 588-606.
observed here might not generalize to other situations in Block, J. (1995). A contrarian view of the five-factor approach to
which personality might be measured (e.g., research pur- personality description. Psychological Bulletin, 117, 187-215.
poses, or solely for development). Care should be taken Block, J. (2001). Millennial contrarianism: The five-factor
then when examining the consistency between these find- approach to personality description 5 years later. Journal of
ings and other studies using data from other contexts. Our Research in Personality, 35, 98-107.
study used ESEM to model facet-level data, and it would be Browne, M. W., & Cudeck, R. (1993). Alternative ways of assess-
useful to apply the same analyses at the item level for the ing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing
NEO-PI-R. Finally, although ESEM has not revealed any structural equation models (pp. 136-162). Beverly Hills, CA:
strong implications for changing applied practice, we have Sage.
only so far examined measurement models (i.e., modeled Byrne, B. (1998) Structural Equation Modeling With Lisrel, Prelis,
responses to the questionnaire itself). It might be that when and Simplis: Basic Concepts, Applications, and Programming
ESEM is used in a broader setting that included criterion (Multivariate Applications Book Series). Lawrence Erlbaum
variables we will identify criteria for which the better fit Associates.
yields predictive improvements. Future research should Chamorro-Premuzic, T. (2007). Personality and individual differ-
investigate this issue. ences. Oxford, England: Blackwell-Wiley.
Because of the unique nature and characteristics of this Chamorro-Premuzic, T., & Furnham, A. (2010). The psychology
sample we also presented descriptive statistics and exam- of personnel selection. Cambridge, England: Cambridge Uni-
ined gender differences and age associations for each of the versity Press.
NEO-PI-R factors and facets. These show that gender dif- Costa, P. T., & McCrae, R. R. (1992). The NEO PI-R professional
ferences are small, and age-related associations are weak. manual. Odessa, FL: Psychological Assessment Resources.
Indeed, it is because past research has shown only small Costa, P., Terracciano, A., & McCrae, R. (2001). Gender differences
demographic associations with personality (Sackett, Schmitt, in personality traits across cultures: Robust and surprising find-
Ellingson, & Kabin, 2001) and still predicts performance ings. Journal of Personality and Social Psychology, 81, 322-331.

Downloaded from asm.sagepub.com at EASTERN KENTUCKY UNIV on November 27, 2014

Furnham et al. 23

Church, A. T., & Burke, P. J. (1994). Exploratory and confirmatory Hu & Bentler’s (1999) findings. Structural Equation Model-
tests of the Big Five and Tellegen’s three- and four-dimensional ing, 11, 320-341.
models. Journal of Personality and Social Psychology, 66, Marsh, H. W., Liem, G. A. D., Martin, A. J., Morin, A. J. S., &
93-114. Nagengast, B. (2011). Methodological measurement fruitful-
De Fruyt, F., Bartels, M., Van Leeuwen, K. G., De Clercq, B., ness of exploratory structural equation modeling (ESEM): New
Decuyper, M., Mervielde, I. (2006). Five types of personality approaches to key substantive issues in motivation and engage-
continuity in childhood and adolescence. Journal of Personal- ment. Journal of Psychoeducational Assessment, 29, 322-346.
ity and Social Psychology, 913, 538-552. Marsh, H. W., Lüdtke, O., Muthén, B., Asparouhov, T.,
DeYoung, C. (2010). Personality neuroscience and the biology of Morin, A. J. S., Trautwein, U., & Nagengast, B. (2010). A new
traits. Social and Personality Psychology Compass, 4, 1165-1180. look at the big-five factor structure through exploratory structural
Digman, J. M. (1997). Higher-order factors of the big five. Jour- equation modeling. Psychological Assessment, 22, 471-491.
nal of Personality and Social Psychology, 73, 1246-1256. McCrae, R. R., & Costa, P. T., Jr. (1987). Validation of the five-
Drasgow, F. (1984). Scrutinizing psychological tests: Measure- factor model across instruments and observers. Journal of
ment equivalence and equivalent relations with external Personality and Social Psychology, 52, 81-90.
variables are the central issues. Psychological Bulletin, 95, McCrae, R. R., Zonderman, A. B., Costa, P. T., Jr., Bond, M. H.,
134-135. & Paunonen, S. V. (1996). Evaluating replicability of factors
Egan, V., Deary, I., & Austin, E. (2000). The NEO-FFI: Emerg- in the revised NEO personality inventory: Confirmatory factor
ing British norms and an item level analysis suggest N, A and analysis versus procrustes rotation. Journal of Personality and
C are more reliable than O and E. Personality and Individual Social Psychology, 70, 552-566.
Differences, 29, 907-920. Millsap, R. E. (1995). Measurement invariance, predictive
Fan, X. (1998). Item response theory and classical test theory: An invariance, and the duality paradox. Multivariate Behavioral
empirical comparison of their item/person statistics. Educa- Research, 30, 577-605.
tional and Psychological Measurement, 58, 357-381. Muthén, L., & Muthén, B. (2006). Mplus user’s guide. Los Angeles,
Ferrando, P., & Chico, E. (2007). The external validity of scores CA: Muthén & Muthén.
based on the two-parameter logistic model: Some compari- Perugini, M., & Gallucci, M. (1997). A hierarchical faceted
sons between IRT and CTT. Psicológica, 28, 237-257. model of the Big Five. European Journal of Personality, 11,
Goldberg, L. R. (1990). An alternative “description of personality”: 279-301.
The Big-Five factor structure. Journal of Personality and Social Rushton, J. P., & Irwing, P. (2009). A General Factor of Person-
Psychology, 59, 1216-1229. ality (GFP) from the Multidimentional Personality Question-
Goldberg, L. R. (1993). The structure of phenotypic personality naire. Personality and Individual Differences, 47, 571-576.
traits. American Psychologist, 48, 26-34. Sackett, P. R., Schmitt, N., Ellingson, J. E., & Kabin, M. B.
Hu, L., & Bentler, P. M. (1995). Evaluating Model Fit. In R.H. (2001). High-stakes testing in employment, credentialing, and
Hoyle (Ed.), Structural Equation Modeling Concepts Issues higher education: Prospects in a post-affirmative action world.
and Applications, (pp. 76-99). Thousand Oaks, CA: Sage. American Psychologist, 56, 302-318.
Hogan, R., & Chamorro-Premuzic, T. (in press). Personality and Saucier, G., & Goldberg, L. R. (1998). What is beyond the Big
career success. In L. Cooper et al. (Eds.). Handbook of person- Five? Journal of Personality, 66, 495-524.
ality and social psychology. Washington, DC: APA. Saucier, G., & Ostendorf, F. (1999). Hierarchical subcomponents of
Horn, J. L., & McArdle, J. J. (1992). A practical and theoretical the Big Five personality factors: A cross-language replication.
guide to measurement invariance for aging research. Experi- Journal of Personality and Social Psychology, 76, 613-627.
mental Aging Research, 18, 117-144. Steiger, J. H. (1990). Structural model evaluation and modifica-
MacCallum, R. (1998). Commentary on quantitative methods in tion. Multivariate Behavioral Research, 25, 173-180.
I/O research. Industrial-Organizational Psychologist, 35, 4. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis
Marsh, H. W., Hau, K. T., & Wen, Z., (2004). In search of golden of the measurement invariance literature: Suggestions, prac-
rules: Comment on hypothesis-testing approaches to setting tices, and recommendations for organizational research. Orga-
cutoff values for fit indexes and dangers in over-generalising nizational Research Methods, 3, 4-70.

Downloaded from asm.sagepub.com at EASTERN KENTUCKY UNIV on November 27, 2014