Judgment and Decision Making, Vol. 15, No. 4, July 2020, pp.

476–498

On the belief that beliefs should change according to evidence:

Implications for conspiratorial, moral, paranormal, political, religious,
and science beliefs
Gordon Pennycook∗, James Allan Cheyne† Derek J. Koehler† Jonathan A. Fugelsang†

Abstract

Does one’s stance toward evidence evaluation and belief revision have relevance for actual beliefs? We investigate the role
of endorsing an actively open-minded thinking style about evidence (AOT-E) on a wide range of beliefs, values, and opinions.
Participants indicated the extent to which they think beliefs (Study 1) or opinions (Studies 2 and 3) ought to change according
to evidence on an 8-item scale. Across three studies with 1,692 participants from two different sources (Mechanical Turk
and Lucid for Academics), we find that our short AOT-E scale correlates negatively with beliefs about topics ranging from
extrasensory perception, to respect for tradition, to abortion, to God; and positively with topics ranging from anthropogenic
global warming to support for free speech on college campuses. More broadly, the belief that beliefs should change according
to evidence was robustly associated with political liberalism, the rejection of traditional moral values, the acceptance of science,
and skepticism about religious, paranormal, and conspiratorial claims. However, we also find that AOT-E is more strongly
predictive for political liberals (Democrats) than conservatives (Republicans). We conclude that socio-cognitive theories of
belief (both specific and general) should take into account people’s beliefs about when and how beliefs should change – that is,
meta-beliefs – but that further work is required to understand how meta-beliefs about evidence interact with political ideology.
Keywords: actively open-minded thinking, belief revision, meta-beliefs

1 Introduction characterized by analogy to that of lawyers than philosophers

(Haidt, 2012; Haidt, 2001) – that is, the function of human
Rational action requires evidence. Given that beliefs inform reason is to form arguments to convince others, as is the
action, beliefs ought to be informed by evidence. A long- goal of lawyers, and not necessarily to form accurate beliefs,
standing broad perspective on human cognition holds that as is the goal of philosophers (Mercier, 2016; Mercier &
reason is, at least to some extent, responsible for accurate be- Sperber, 2011). Of course, the analogy does not imply that
lief formation (Baron, 2008; Kohlberg, 1969; Piaget, 1932; people only reason like lawyers or like philosophers, but
Stanovich, 2005). However, the human capacity to revise rather that the typical characteristics of human cognition are
beliefs in the face of conflicting evidence is, charitably, im- more similar to one frame of thinking than the other. To
perfect. Humans are prone to motivated reasoning (Kunda, simplify, some researchers have disputed the common idea
1990), identity protective cognition (Kahan et al., 2012), that reasoning facilitates sound judgment by pointing to cases
confirmation bias (Nickerson, 1998), myside bias (Perkins, (e.g., motivated reasoning) where explicit reasoning actually
2019; Stanovich, West & Toplak, 2013), naïve realism (Ross hurts judgment (Kahan et al., 2012; Kahan, Peters, Dawson
& Ward, 1996), and bias blind spots (Pronin, Lin & Ross, & Slovic, 2017).
2002). There is widespread disagreement about the role and Relatedly, given evidence that we rely heavily on a num-
consequences of the human capacity to reason. ber of heuristics and biases (Kahneman, Slovic & Tversky,
Various analogies have been used to simplify the various 1982) and that unconscious processes have an (apparently)
broad perspectives on human thought and, although they may widespread impact on our decisions (Bargh & Chartrand,
be oversimplifications, they illustrate the disagreement. For 1999), a prominent perspective is that explicit reasoning
example, it has been argued that human reasoning is better and deliberation is just not very effective in the context of
powerful intuitions (e.g., Bargh, 1999; Bargh, Schwader,
Hailey, Dyer & Boothby, 2012; Dijksterhuis & Strick, 2016;
Preparation of this manuscript was supported by the Social Sciences
and Humanities Research Council of Canada and the Natural Sciences and Gigerenzer, 2007; Haidt, 2001). One famous analogy is that
Engineering Research Council of Canada. human cognition is like an emotional (or intuitive) dog with
Copyright: © 2020. The authors license this article under the terms of a rational tail (Haidt, 2001) (or, in a more recent analogy, an
the Creative Commons Attribution 3.0 License.
∗ Hill/Levene Schools of Business, University of Regina, Canada. Email: intuitive elephant and an analytic rider; Haidt, 2012): That
[email protected]. is, our capacity to reason does not effectively override our
† Department of Psychology, University of Waterloo, Canada. intuitions and emotional impulses.

476
https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press
 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 477

These three perspectives can be summarized simplisti- also evidence that analytic thinking is associated with dis-
cally in terms of three general claims about the nature of hu- belief in a variety of epistemically suspect beliefs (Penny-
man reasoning: 1) that reasoning prototypically helps make cook et al., 2015a), such as in paranormal and religious be-
good decisions and come to informed beliefs (“reasoning liefs (Pennycook, Cheyne, Seli, Koehler & Fugelsang, 2012;
is helpful”); 2) that reasoning is prototypically ineffective, Pennycook, Ross, Koehler & Fugelsang, 2016; Shenhav,
since intuition dominates human cognition (“reasoning is Rand & Greene, 2012), conspiratorial ideation (Swami, Vo-
helpless”); and 3) that reasoning prototypically undermines racek, Stieger, Tran & Furnham, 2014), anti-science beliefs
sound judgment and exacerbates motivated reasoning and (and specifically rejection of evolution) (Gervais, 2015), and
(for example) political polarization (“reasoning is hurtful”). pseudo-profound bullshit receptivity (Pennycook, Cheyne,
Although any of the three accounts may be the best ex- Barr, Koehler & Fugelsang, 2015). In addition, there is evi-
planation for the underlying psychology behind any given dence that reliance on intuition is associated with traditional
belief/opinion/value, the critical question here is which ac- moral values (Pennycook, Cheyne, Barr, Koehler & Fugel-
counts offers the best broad description of high-level human sang, 2014; Royzman, Landy & Goodwin, 2014) and conser-
cognition (i.e., which has the greatest explanatory power vative political ideology (Jost, 2017) (but perhaps more-so
across various beliefs/opinions/values). with political apathy; see Pennycook & Rand, 2019a).
Although recent work has attempted to mediate between In contrast, a large and diverse body of evidence sup-
these three broad accounts by investigating individual dif- ports the idea that reason is perhaps overrated. For instance,
ferences in analytic thinking (e.g., Pennycook, Fugelsang intuitive heuristics are often extremely useful and, in some
& Koehler, 2015a; Pennycook & Rand, 2019b; Pennycook, contexts, may actually be more accurate than reasoned reflec-
2018), this work is vague on the specific aspects of analytic tion (Gigerenzer, 2007; Gigerenzer, Todd & ABC Research
thinking that support good thinking. Here we contend that Group, 1999). Illustrative (albeit extreme) examples of this
people differ in terms of their explicit stance toward whether come from research on expertise (Klein, 2008), which shows
beliefs ought to change according to evidence and that this that naturalistic decision making (e.g., among chess masters
has major consequences for what beliefs, opinions, and val- or firefighters) allows for very rapid yet extremely accurate
ues that they hold. That is, some may place stronger value choice (Kahneman & Klein, 2009). Furthermore, social
in changing their beliefs and taking relevant evidence into psychology in the 1990’s provided numerous examples of
account – and thereby (for example) take a stance toward the surprising power of intuition (Bargh & Chartrand, 1999;
reasoning that is more akin to a philosopher – whereas oth- Dijksterhuis & Strick, 2016; Haidt, 2012). Although there
ers may place stronger value in maintaining constancy and have been questions about the replicability of some of these
the defense of prior beliefs – and thereby take a stance to- effects (e.g., for so-called “social priming” [Cesario, 2014]),
ward reasoning that is more akin to a lawyer. Moreover, the strong influence of intuition on decision making is not a
this meta-belief may impact what sort of beliefs individuals matter of dispute (Evans, 2008; Kahneman & Klein, 2009).
endorse as adults, indicating that reasoning really does have
an impact on intuitive beliefs. The goal of the present work 1.2 Does reasoning undermine sound judg-
is to investigate these possibilities.
ment?
1.1 Is reasoning helpful or helpless? In contrast to the work just reviewed, there is also consider-
able evidence for motivated reasoning effects (Kunda, 1990)
Dual-process theories of reasoning, which distinguish from – that is, cases where reasoning actively hurts sound judg-
autonomous (intuitive) processes and those that are accom- ment and causes people to become further entrenched in
plished via some form of deliberative control (De Neys, what they already believe (Kahan, 2013). For example, peo-
2017; Evans & Stanovich, 2013; Pennycook, Fugelsang ple tend to dismiss information that is inconsistent with their
& Koehler, 2015b; Thompson, Prowse Turner & Penny- political ideology (Lodge & Taber, 2005; Redlawsk, 2002;
cook, 2011), typically emphasize how controlled reasoning Redlawsk, Civettini & Emmerson, 2010; Strickland, Taber
processes can override (sometimes) incorrect intuitive re- & Lodge, 2011) and engage in biased search for information
sponses. Although this emphasis does not imply that rea- that is supportive of their beliefs (i.e., confirmation bias;
soning and accuracy are synonymous, it does suggest that Knobloch-Westerwick, Mothes & Polavin, 2017; Nickerson,
there are meaningful and important cases where such an 1998). In fact, there is evidence that political polarization
association is present (Evans, 2012). To take a recent ex- about contentious scientific issues (such as global warming)
ample, individuals who are more disposed toward thinking is actually greater among individuals who are more intelli-
analytically (as indexed by the Cognitive Reflection Test; gent (Kahan et al., 2012; Kahan, Peters, Dawson & Slovic,
Frederick, 2005) are less likely to fall for fake news regard- 2017; Sarathchandra, Navin, Largent & McCright, 2018)
less of whether it is consistent or inconsistent with their and who report having a more actively open-minded think-
political ideology (Pennycook & Rand, 2019b). There is ing style (Kahan & Corbin, 2016; but see Baron, 2017).

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 478

A parsimonious broad account of these findings is that berg & Weisberg, 2018). Indeed, the actively open-minded
individuals engage analytic reasoning processes, not in the thinking scale (AOT) was created to assess (in part) the be-
service of accuracy, but as a means to protect their identity lief that it is good to seek evidence that may conflict with
(Kahan, 2013) or to form convincing arguments (Mercier, intuitions (Baron, 2008; Baron et al., 2015; Baron, 1985;
2016). This perspective flips the common conception of Stanovich & West, 1997; see also Price, Ottati, Wilson &
human reasoning on its head and suggests that reasoning of- Kim, 2015, for a measure based more on self-report) – a
ten makes people more unreasonable. Consistent with this tendency that is associated with improved decision making
account, a recent meta-analysis indicated that partisan bias over and above intelligence or cognitive ability (Stanovich &
effects (motivated reasoning) were equivalent across the po- West, 2000; Stanovich & West, 1998). Moreover, much like
litical spectrum (Ditto et al., 2019; but see Baron & Jost, individual differences in cognitive reflection, high AOT has
2019). This research indicates that reasoning is typically been linked to skepticism about supernatural claims (Baron
(or, at least, frequently) used in service of justifying prior et al., 2015; Pennycook, Cheyne, Barr, Koehler & Fugel-
beliefs, as opposed to updating them based on the evidence sang, 2014; Svedholm & Lindeman, 2013) and superstition
presented. To investigate this issue, we will focus on the (Sá, West, & Stanovich, 1999), indicating that the AOT scale
idea that individuals who are more prone to engage in rea- may index some aspects of openness to evidence in belief
soning are more (not less) politically polarized. Consistent formation and revision. Indeed, Svedholm-Häkkinen and
evidence for increased polarization among highly reflective Lindeman (2018) found a “fact resistance” factor within the
people would indeed indicate that motivated reasoning is to broader AOT measure that consists of items that ask about
be expected; to return to an earlier analogy, that humans beliefs about changing beliefs according to evidence. As
reason more like lawyers than philosophers. noted by Baron (2019), it is this “flexible thinking” dimen-
sion that is most central to the concept of AOT. Shortened
versions of the AOT scale have also typically focused largely
1.3 Actively open-minded thinking
on the belief revision questions (Baron et al., 2015; Haran,
Despite research showing evidence for motivated reasoning Ritov & Mellers, 2013), which further suggests that these
and the power of intuitions, the previously reviewed associ- items are of particular relevance for the AOT’s predictive
ations between analytic thinking and various beliefs/values validity. Nonetheless, the broad consequences of this meta-
suggests that reasoning is nonetheless used to modify beliefs belief across a variety of domains has not yet been systemat-
in everyday life (although other factors are of course involved ically investigated despite having major relevance for several
in determining what people believe). That is, people who are broad theories of human cognition. We will refer to our
more reflective when they are given a trick question from the subscale simply as actively open-minded thinking about evi-
Cognitive Reflection Test (CRT) have different beliefs than dence (AOT-E). The items for our scale can be found in Table
intuitive people. A parsimonious explanation of this is that 1. Our AOT-E scale is not the same as has been used in the
the same people who reflect on the CRT also tend to reflect past, although some of the items are from previous (longer)
about their beliefs (i.e., they use reason to modify beliefs). versions of the AOT. For further information on how we de-
Nonetheless, the disposition to engage analytic thinking is rived the AOT-E scale from the larger full AOT scale, see
not the same as having an actively open-minded stance in the two validation studies presented in the supplementary
general (Baron, 1985; Stanovich & West, 1997) or toward materials.
evidence in particular (Baron, 2019; Baron, Scott, Fincher
& Metz, 2015). Indeed, analytic thinking may be used to
1.4 Current work
both override intuitions (i.e., to modify or undermine prior
beliefs) or to rationalize or bolster intuitions (i.e., to reinforce Is reasoning prototypically helpful, helpless, or hurtful? One
prior beliefs) (Pennycook, Fugelsang, et al., 2015b). possibility that has not yet been broached is that the three
In the present work, we will focus instead on people’s perspectives are primarily describing different people. That
beliefs about whether beliefs and opinions should change ac- is, people have different beliefs about whether beliefs should
cording to evidence. Moreover, we will investigate a wide change according to evidence (“meta-beliefs”) and this has
range of beliefs, values, and opinion together as a way to sys- consequences for the effectiveness of their reasoning and,
tematically assess the potential long-term impact of people’s therefore, what types of beliefs that they hold. The goal
thinking style on what they think. of the present investigation is to determine whether AOT-
The idea that some people may not be disposed to using ev- E is correlated with as wide a variety of beliefs, values,
idence to inform their beliefs has been broached previously. and opinions as is feasible in a single study. If AOT-E is
For example, people may differ in their “criteria” for belief; consequential, it should be associated with people’s stances
although some hold that evidence and scientific consensus on a number of important issues. To this end, we investigated
are most important, others believe that “knowledge of the conspiratorial, moral, paranormal, political, religious, and
heart” should also be a central consideration (Metz, Weis- science beliefs.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 479

Table 1: The Actively Open-minded Thinking about Evidence (AOT-E) scale. Items 3, 4, 5, 7 & 8 are reverse scored.

# Item AOT Subscale

1 A person should always consider new possibilities. AOT
2 People should always take into consideration evidence that goes against their beliefs. AOT
3 It is important to persevere in your beliefs even when evidence is brought to bear against them. Belief Identification
(rev)
4 Certain beliefs are just too important to abandon no matter how good a case can be made against Belief Identification
them. (rev)
5 One should disregard evidence that conflicts with your established beliefs. (rev) Belief Identification
6 Beliefs should always be revised in response to new information or evidence. Belief Identification
7 No one can talk me out of something I know is right. (rev) Dogmatism
8 I believe that loyalty to one’s ideals and principles is more important than “open-mindedness”. Openness-Values
(rev)

2 Study 1 secret group of people is responsible for making all major

world decisions, such as going to war” (𝛼 = .97). Responses
2.1 Method were made on the following 5-point scale: 1) Definitely not
true, 2) Probably not true, 3) Not sure/cannot decide, 4)
2.1.1 Participants
Probably true, 5) Definitely true.
American participants were recruited from Mechanical Turk
on February 18th , 2016. We set our goal sample at 350
and over sampled 380 participants (assuming some degree Paranormal belief. Participants completed a slightly re-
of attrition due to random responding). Only 3 participants vised Paranormal Belief Scale (Pennycook, Cheyne, Seli,
responded affirmatively when asked if they responded ran- Koehler & Fugelsang, 2012; Tobacyk, 2004) with the reli-
domly at any point during the survey and 3 participants did gious belief items excluded (𝛼 = .95). The scale consisted
not answer affirmatively when asked if they are fluent in En- of 22 items sampled from 6 categories of supernatural be-
glish. The resulting sample (N = 375, Mean age = 35.8) lief (example items in parentheses): Psi (“Mind reading is
consisted of 216 males and 158 females (1 participant did possible”), Witchcraft (“Witches do exist”), Omens of luck
not indicate their gender). (“Black cats can bring bad luck”), Spiritualism (“It is pos-
sible to communicate with the dead”), Extraordinary life
2.1.2 Materials forms (“The Loch Ness monster of Scotland exists”) and
Precognition (“Astrology is a way to accurately predict the
Measures were converted into POMP scores, i.e.(raw- future”). Participants indicated their belief by responding on
min)/(max-min), ranging from 0-100 (Cohen, Cohen, Aiken a 7-point scale from 1) Strongly disagree, to 4) Uncertain, to
& West, 1999). Data and materials for all studies are avail- 7) Strongly agree.
able on OSF: https://osf.io/xqzse/.

AOT-E. We administered the AOT-E scale that is pre- God Skepticism. Skepticism about God was assessed us-
sented in Table 1. Participants responded on a scale from ing the following question: “What sort of God, if any, do you
1) Strongly disagree to 6) Strongly agree. The AOT-E had believe in?” and presenting the following options of increas-
strong reliability (𝛼 = .87). Participants rated themselves ing skepticism (Pennycook et al., 2012; Pennycook, Ross, et
as, on average, willing to change their beliefs according to al., 2016): 1) A personal God [Theism], 2) God as an imper-
evidence (M = 69.8, SD = 19.1 – scale ranges from 0–100). sonal force [Pantheism], 3) A God who created everything,
Only 19.2% of the participants were at or below the scale but does not intervene in human affairs [Deism], 4) Don’t
midpoint (indicating a resistance to evidence). know whether or not any Gods exist [Negative Agnostic], 5)
Don’t know whether or not any Gods exist and no one else
Conspiracist ideation. Participants completed a 15-item does either [Positive Agnostic], 6) I don’t believe in Gods of
general conspiracy beliefs scale (Brotherton, French & Pick- any sort [Negative Atheist], and 7) I believe that God does
ering, 2013). The scale included items such as “A small, not exist [Positive Atheist].

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 480

Table 2: Political opinions scale, Study 1. Items are labeled with (rev) if they were subsequently reverse-scored (for our
analysis) so that a higher score indicates a more conservative belief. Conservatism is the mean of the social and economic
conservative political ideology questions. Mean scores range from 0–100, with 0 indicating complete disagreement with the
statement and 100 indicating complete agreement.

Name Item Correlation (r) Mean (SD)

with conservatism
Abortion Abortion should be legally available with few or no restrictions. −.59 67.3 (35.1)
(rev)
Same Sex Marriage People of the same sex should be permitted to marry. (rev) −.53 78.0 (32.7)
Military One of the most important things a government can do is to make .52 44.8 (31.5)
sure that its military forces remain strong, even if social programs
have to be cut back.
War It is better for a country to be ready to go to war than to be pushed .42 43.3 (30.7)
around or to waste time in negotiations.
Police Authority There is too much focus on the rights of suspects, because police .40 35.1 (30.0)
need more authority to deal with criminals.
Men in Feminism Is there room for men in feminism? (rev) −.39 67.7 (28.8)
Capital Punishment It is important to have the death penalty available as a deterrent to .38 53.6 (36.3)
very serious crimes such as murder.
Sexism for Men Men experience sexism on par with women. .24 39.7 (30.2)
Microaggressions Microaggressions are a serious problem in educational contexts −.19 53.4 (28.5)
(such as in universities) 𝜏 . (rev)
Campus Free Speech Students should be able to block controversial speakers from −.08 33.0 (30.6)
giving talks at their university. (rev)
𝜏 The following note was also presented: Microaggressions are defined as “brief, everyday exchanges that send denigrating
messages to certain individuals because of their group membership.”
All corrlations are significant at 𝑝 < .001 except the last, which is n.s.

Moral values. We used Pennycook, Cheyne, Barr, nomic issues, 2) Consistent Conservatives, who are conser-
Koehler, and Fugelsang’s (2014) moral values scale, which vative/very conservative on both social and economic issues,
consisted of 6 care/fairness (“individualising”) and 4 tradi- 3) Libertarians, who are liberal/very liberal on social issues
tional (“binding”) moral values (Graham et al., 2011). Par- but conservative/very conservative on economic issues, and
ticipants were asked to rate how important the values were to 4) Consistent Moderates who are moderate on both social
their moral thinking on a 7-point scale from 1) Irrelevant to and economic issues. However, because of the liberal politi-
7) Extremely Important. Care/fairness values included be- cal skew of Mechanical Turk, there were only 60 Consistent
ing kind, supporting the autonomy of others, being helpful, Conservatives in our sample compared to 153 Consistent
being fair, avoiding harm, and supporting the rights of oth- Liberals.
ers (𝛼 = .85). Traditional values included showing respect
for traditions, being patriotic and loyal, showing respect for Political opinions. We also surveyed a range of political
legitimate authority, and being pure by avoiding carnal plea- opinions (see Table 2). Participants were asked to indicate
sures and disgusting things (𝛼 = .80). agreement/ disagreement on a 7-point scale from 1) Strongly
disagree to 7) Strongly agree. As is evident from Table 2,
Political ideology. Participants were asked to indicate their three of the items did not correlate particularly highly with
stance on social and economic issues separately on scales political ideology (microaggressions, campus free speech
from 1) Very liberal, to 3) Moderate, to 5) Very conserva- [coded so that support of free speech was counted as conser-
tive. Following Pennycook and Rand (2019a), we computed vative], and men experiencing sexism). We therefore created
four political categories based on the convergence between a Conservative Opinions scale (𝛼 = .81) using all items ex-
social and economic political ideology: 1) Consistent Lib- cept for these three (all items were re-scored so that a high
erals, who are liberal/very liberal on both social and eco- score indicated a more conservative opinion). Participants

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 481

Table 3: Science beliefs scale, Study 1. Items are labelled with (rev) if they were reverse scored so that a higher score
indicates a more pro-scientific belief. Conservatism is the mean of the social and economic conservative political ideology
questions. In every case, political conservatives held a more anti-scientific stance. Mean scores range from 0-100, with 0
indicating complete disagreement with the statement and 100 indicating complete agreement. (All correlations are significant
at 𝑝 < .001.)

Name Item Correlation (r) Mean (SD)

with conservatism
Global warming Global warming is at least partly caused by human activity and is a −.55 80.6 (27.5)
serious problem for the environment.
Evolution Evolution is the best explanation so far for our origins. −.43 73.7 (31.0)
Big Bang The big bang theory is, generally speaking, the best explanation we −.39 70.6 (31.3)
have so far for the origin of our universe.
Stem Cells Stem cell research is a productive enterprise. −.32 77.0 (25.2)
Vaccines/Autism Vaccines can cause autism in children. (rev) −.31 26.0 (29.3)
Old Earth The universe is billions of years old. −.29 85.5 (23.1)
Modern Medicine Modern medicine is the most effective means of treating most disease. −.17 78.4 (24.0)
Tech Problems Technology causes more problems than it helps solve. (rev) −.16 29.3 (27.7)
GMO/Health Genetically modified foods are hazardous to human health. (rev) −.16 51.7 (31.3)

were also asked to indicate their relative trust in the govern- responded on a 7-point scale from 1) Strongly disagree to
ment on a scale from 1) Strongly Distrust to 5) Strongly Trust 7) Strongly agree; however, for our primary analysis, all
(this was also uncorrelated with political conservatism, r = items were scored such that a higher value meant a more
-.07). pro-science belief. The full scale had good reliability, 𝛼 =
.84. Participants were also asked to indicate their relative
trust in scientists on a scale from 1) Strongly Distrust to 5)
Free Market Ideology. Participants completed a 5-item
Strongly Trust.
Free Market Ideology measure (Heath & Gifford, 2006). The
scale assesses the belief in the powers of the free market (𝛼 =
.85). It includes items such as: “An economic system based Demographics. Participants were given a demographic
on free markets unrestrained by government interference au- questionnaire that included the following items: age, gen-
tomatically works best to meet human needs.” Responses der, and English proficiency. Social and economic political
were provided on a 7-point scale from 1) Strongly disagree ideology were included in the demographics questionnaire.
to 7) Strongly agree.
2.1.3 Procedure
Science beliefs. We created a science belief scale based on Participants either completed the AOT-E at the beginning
various contemporary scientific issues. In particular, we se- of the survey or at the end (but before demographics). The
lected a number of typical science-related beliefs (Table 3): presentation order did not change the pattern of results and
evolution, anthropogenic global warming, big bang theory, the aggregate results will therefore be reported. Otherwise,
old Earth, and stem cell research. We also attempted to use the order of the measures was as follows: 1) conspiracist
items that have been associated with “liberal” anti-science at- ideation, 2) paranormal belief, 3) moral values, 4) science
titudes (Table 3): resistance to technology (reverse scored), beliefs, 5) political opinions, 6) free market ideology, 7)
genetically modified organism (GMO) resistance (reverse theism, and 8) demographics (including political ideology).
scored), vaccines as a cause of autism (reverse scored), and
belief in modern medicine. However, as is evident from Ta-
2.2 Results and Discussion
ble 3, political conservatives were more likely to hold the
more anti-scientific stance on every single issue – even is- As is evident from Table 4, AOT-E was strongly associated
sues often associated with political liberalism. Nonetheless, with every other primary measure. Individuals who believe
consistent with prior research, there was large variability in that beliefs should change according to evidence (those high
terms of how strongly conservatism predicted anti-scientific in AOT-E) were: a) less likely to believe conspiratorial, para-
attitudes (Rutjens, Sutton & van der Lee, 2018). Participants normal, and religious (and, specifically, theistic) claims, b)

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 482

Table 4: Correlations (Pearson r) among primary measures in Study 1 (Mechanical Turk). Cronbach’s Alpha for each scale
is listed in brackets along the major diagonal. AOT-E = Actively Open-minded Thinking about Evidence. MV = Moral Values.
N = 375.

1 2 3 4 5 6 7 8 9 10 11
1. AOT-E (.87)
2. Conspiracy Beliefs −.36∗∗∗ (.95)
3. Paranormal Beliefs −.40∗∗∗ .63∗∗∗ (.97)
4. Traditional MV −.54∗∗∗ .16∗∗ .27∗∗∗ (.80)
5. Care/Fairness MV .26∗∗∗ −.10 −.09 .12∗ (.85)
6. Social Conservatism −.46∗∗∗ .20∗∗∗ .18∗∗ .46∗∗∗ −.16∗∗ ...
7. Fiscal Conservatism −.24∗∗∗ .11∗ .12∗ .27∗∗∗ −.11∗ .69∗∗∗ ...
8. Free Market Ideology −.34∗∗∗ .13∗ .11∗ .29∗∗∗ −.18∗∗ .45∗∗∗ .53∗∗∗ (.85)
9. Conservative Opinions −.61∗∗∗ .28∗∗∗ .30∗∗∗ .64∗∗∗ −.23∗∗∗ .72∗∗∗ .54∗∗∗ .55∗∗∗ (.79)
10. Pro-Science Beliefs .65∗∗∗ −.52∗∗∗ −.49∗∗∗ −.47∗∗∗ .17∗∗ −.54∗∗∗ −.32∗∗∗ −.31∗∗∗ −.63∗∗∗ (.84)
11. God Skepticism .52∗∗∗ −.28∗∗∗ −.48∗∗∗ −.48∗∗∗ .02 −.37∗∗∗ −.24∗∗∗ −.19∗∗∗ −.47∗∗∗ .55∗∗∗ ...
***indicates p < .001, **indicates p < .01, *indicates p < .05.

less likely to hold traditional moral values but were more with disagreement, r = .07)1; and whether students should
likely to adopt care/fairness moral values, c) less conserva- be able to block controversial speakers from giving talks
tive in terms both economic and fiscal ideology (including at their university (AOT-E was significantly associated with
free market ideology) and across a range of specific politi- disagreement, r = .25). The items most strongly associated
cal opinions, and d) less likely to hold anti-science beliefs. with political conservatism were most strongly negatively
Gignac and Szodorai (2016) meta-analyzed typical effect associated with AOT-E.
sizes across social psychology and found that correlations The pattern of results for individual science belief items
(r) of .10, .20, and .30 can be considered relatively small, (Table 6) was very clear (and plainly in support of the “rea-
medium, and large, respectively. Using this metric, AOT-E soning helps” perspective): AOT-E was associated with
was a remarkably strong predictor of most factors. With the more agreement with scientists, regardless of whether the
exceptions of care/fairness moral values (r = .26) and the issue pertained to agreement with a clear scientific consen-
single-item fiscal conservatism (r = −.24), every effect size sus (such as around anthropogenic global warming or the big
was above what would be considered large based on em- bang) or a disagreement with an anti-scientific belief (such
pirical norms. The correlation with conservative opinions as that GMO’s are unhealthy or that vaccines cause autism).
and pro-science beliefs, in particular (r’s greater than .60), AOT-E was also positively associated with general trust in
were well above the 95th percentile (r = .45) in terms of scientists.
effect size norms for individual differences research in psy-
chology (Gignac & Szodorai, 2016). This overall pattern of
result undermines the idea that reasoning is ineffective and 3 Study 2
is consistent with the general claim that reasoning has major
The results of Study 1 indicate that AOT-E is a very strong
impacts on our beliefs and values.
predictor of a wide range of beliefs and opinions. There are,
To further understand the scope of AOT-E’s predictive va-
however, three key issues that the data from Study 1 leave
lidity, we also investigated the extent to which it predicted
unresolved. The first pertains to the perhaps implausibly
specific political opinions (Table 5). With respect to political
large effect sizes that we found in Study 1. A recent paper by
opinions, individuals who indicate being more actively open- Stanovich and Toplak (2019) raised an important point that
minded about evidence held broadly liberal political views. pertains to the AOT (and that applies to the AOT-E): When
Indeed, AOT-E was less predictive for the items that were asked about “beliefs”, some individuals may assume that the
less strongly associated with political ideology: whether
men experience sexism on par with women (AOT-E was sig- 1We subsequently realized that the wording of this item may be prob-
lematic: Individuals might indicate that microaggressions are a problem
nificantly associated with disagreement, r = −.24); whether
because people are being microaggressed, but others might indicate that
microaggressions are a serious problem in educational con- they are a problem because people are (falsely, presumably) indicating that
texts (AOT-E was slightly but non-significantly associated they are being microaggressed.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 483

Table 5: Correlations (Pearson r) between AOT-E and political opinion items in Study 1 (Mechanical Turk). Opinion items
are scored such that a higher score corresponds with a more strongly politically conservative position and are organized in
order of the strength of positive association with conservatism (see Table 2). AOT-E = Actively Open-minded Thinking about
Evidence. N = 375.

1 2 3 4 5 6 7 8 9 10 11
1. AOT-E
2. Abortion −.43∗∗∗
3. Same Sex Marriage −.50∗∗∗ .63∗∗∗
4. Military −.40∗∗∗ .40∗∗∗ .32∗∗∗
5. War −.37∗∗∗ .26∗∗∗ .29∗∗∗ .62∗∗∗
6. Police Authority −.51∗∗∗ .35∗∗∗ .43∗∗∗ .48∗∗∗ .44∗∗∗
7. Men in Feminism −.37∗∗∗ .38∗∗∗ .47∗∗∗ .30∗∗∗ .28∗∗∗ .33∗∗∗
8. Capital Punishment −.33∗∗∗ .15∗∗ .23∗∗∗ .46∗∗∗ .43∗∗∗ .46∗∗∗ .19∗∗∗
9. Sexism for Men −.24∗∗∗ .12∗ .10∗ .30∗∗∗ .23∗∗∗ .21∗∗∗ .22∗∗∗ .22∗∗∗
10. Microaggressions .06 .04 .002 .07 −.04 −.07 .14∗∗∗ .001 .17∗∗∗
11. Campus Free Speech 𝜏 .25 −.01 −.08 −.03 −.03 −.17∗∗
∗∗∗ .05 −.08 −.06 .24∗∗∗
12. Trust in Government −.23∗∗∗ −.004 .03 .10 .08 .27∗∗∗ .01 .09 .13∗ −.05 −.22∗∗∗
𝜏 Campus free speech was not significantly associated with conservatism (see Table 2). At any rate, a high score indicates
opposition to the idea that “students should be able to block controversial speakers from giving talks at their university”.
***indicates p < .001, **indicates p < .01, *indicates p < .05.

Table 6: Correlations (Pearson r) between AOT-E and science belief items in Study 1 (Mechanical Turk). Items are scored
such that a higher score corresponds with a more strongly pro-science stance and are organized in order of the strength
of negative association with conservatism (see Table 3). AOT-E = Actively Open-minded Thinking about Evidence. GMO =
Genetically Modified Organisms. N = 375.

1 2 3 4 5 6 7 8 9 10 11
1. AOT-E −
2. Global warming .37∗∗∗ −
3. Evolution .51∗∗∗ .38∗∗∗ −
4. Big Bang .51∗∗∗ .34∗∗∗ .78∗∗∗ −
5. Stem Cells .45∗∗∗ .31∗∗∗ .55∗∗∗ .54∗∗∗ −
6. Vaccines/Autism .47∗∗ .26∗∗∗ .39∗∗∗ .37∗∗∗ .40∗∗∗ −
7. Old Earth .40∗∗∗ .33∗∗∗ .47∗∗∗ .45∗∗∗ .41∗∗∗ .38∗∗∗ −
8. Modern Medicine .33∗∗∗ .18∗∗∗ .36∗∗∗ .37∗∗∗ .47∗∗∗ .53∗∗∗ .33∗∗∗ −
9. Tech Problems .44∗∗∗ .15∗∗ .28∗∗∗ .28∗∗∗ .36∗∗∗ .49∗∗∗ .21∗∗∗ .47∗∗∗ −
10. GMO/Health .36∗∗∗ .06 .33∗∗∗ .37∗∗∗ .34∗∗∗ .52∗∗∗ .24∗∗∗ .42∗∗∗ .39∗∗∗ −
11. Trust Scientists .35∗∗∗ .42∗∗∗ .48∗∗∗ .49∗∗∗ .47∗∗∗ .43∗∗∗ .29∗∗∗ .43∗∗∗ .33∗∗∗ .31∗∗∗ −
*** indicates p < .001, ** indicates p < .01, *indicates p < .05.

question is really about religious beliefs. Indeed, Stanovich be partly due to the possibility that AOT-E plays a major
and Toplak found that the extremely high correlation between role in belief formation (as we have argued). However, it
AOT and religious beliefs can be partially (but not fully) may also be the case that religious believers are particularly
accounted for using “belief revision” items (that is, the same opposed to revising their religious beliefs, but less opposed
class of items that make up the AOT-E). Of course, this may to revising beliefs in general. Consistent with both of these

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 484

Table 7: A revised Actively Open-minded Thinking about Evidence (AOT-E) scale that asks about “opinions” instead of
“beliefs”.

Original AOT-E Revised AOT-E

A person should always consider new possibilities. A person should always consider new information.
People should always take into consideration evidence that People should always take into consideration evidence that
goes against their beliefs. goes against their opinions.
It is important to persevere in your beliefs even when It is important to persevere in your opinions even when
evidence is brought to bear against them. (rev) evidence is brought to bear against them. (rev)
Certain beliefs are just too important to abandon no matter Certain opinions are just too important to abandon no
how good a case can be made against them. (rev) matter how good a case can be made against them. (rev)
One should disregard evidence that conflicts with your One should disregard evidence that conflicts with your
established beliefs. (rev) established opinions. (rev)
Beliefs should always be revised in response to new Opinions should always be revised in response to new
information or evidence. information or evidence.
No one can talk me out of something I know is right. (rev) It is possible for someone to convince me to change my
mind.
I believe that loyalty to one’s ideals and principles is more I believe that loyalty to one’s ideals and principles is more
important than “open-mindedness”. (rev) important than “open-mindedness”. (rev)

possibilities, Stanovich and Toplak found that items that used assess the relative predictive strength of AOT-E relative to
slightly different wording (which did not lead to the religious CRT.
belief presumption) continued to predict religious belief, but
not as strongly. In Study 2, we therefore modified the AOT-E
to ask about “opinions” instead of “beliefs” (see Table 7). We 3.1 Method
also changed the wording of an additional item so that there
would be an equal number of standard and reverse-coded 3.1.1 Participants
items. Participants in Study 2 were either administered the American participants were recruited from Lucid for Aca-
original AOT-E or the revised AOT-E. Our goal was to ask demics on April 19th , 2019. We recruited 700 participants,
whether the results of Study 1 are robust to variations in who were randomly assigned to one of two conditions. In
AOT-E scale wordings even if effect sizes vary somewhat. total, 751 participants began the study but 60 did not finish.
A second drawback of Study 1 is that our sample came We also removed individuals who responded affirmatively
from Mechanical Turk and is therefore particularly unrepre- when asked if they responded randomly at any point during
sentative of political conservatives. This is a notable draw- the survey (77 from the original AOT-E condition and 76
back because the association between AOT-E and political from the revised AOT-E condition). The resulting sample
opinions may differ depending on whether the individuals (N = 539, Mean age = 45.4) consisted of 251 males and
are politically liberal or conservative. As such, in our sec-
278 females, 1 transgender female, 1 transgender male, 3
ond study, we collected a sample from Lucid for Academics
trans/non-binary, 4 “not listed”, and 1 who preferred not to
– a source that provides American samples that are nation-
answer.
ally representative on age, gender, ethnicity, and geography
(based on quota-matching), and that therefore provides a
more even and representative split of liberals and conser- 3.1.2 Materials
vatives (Coppock & Mcclellan, 2019; Pennycook & Rand,
2019a). Measures identical to Study 1. The following measures
Third, many of the AOT-E correlates reported in Study were administered as in Study 1: Conspiracist ideation, para-
1 have, in previous research, been shown to correlate with normal belief, God skepticism, moral values, political ide-
performance on the Cognitive Reflection Test (CRT; Freder- ology, political opinions, free market ideology, and trust in
ick, 2005; Pennycook, Fugelsang, et al., 2015a) – a measure scientists. Unlike in Study 1, all of the political opinion items
intended to assess the broad disposition to think analyti- were significantly associated with political ideology (Table
cally and that also correlates with AOT (Toplak, West & 8). We therefore used all of the items to form the political
Stanovich, 2011). Thus, in Study 2 we included the CRT to opinions scale (𝛼 = .72).

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 485

Table 8: Political opinions scale, Study 2. Items are labeled soning (Campitelli & Gerrans, 2014; Pennycook, Cheyne,
with (rev) if they were subsequently reverse-scored (for our Koehler & Fugelsang, 2016; Toplak et al., 2011). The Lucid
analysis) so that a higher score indicates a more conservative sample had particularly low accuracy on the CRT (M = .16,
belief. Raw (prior to reverse scoring) correlations with con- SD = .28; i.e., 0.5 out of 3 correct, on average – 70% of the
servatism are presented below. Conservatism is the mean sample got 0 out of 3). As a consequence, reliability was
of the social and economic conservative political ideology relatively low for the CRT (𝛼 = .64).
questions. Mean scores range from 0–100, with 0 indicating
complete disagreement with the statement and 100 indicat-
Religious belief. In addition to the theism measure used
ing complete agreement. ( 𝑝 < .001 for all correlations.)
in Study 1, we also included a full religious belief scale (via
Name Correlation (r) Mean (SD) Pennycook et al., 2016). For this, participants were asked to
with conservatism indicate their degree of belief in the following supernatural
religious claims: afterlife, heaven, hell, miracles, angels,
Abortionrev −.46∗∗∗ 54.1 (37.3) demons, soul, devil/Satan, and God. Participants responded
Same Sex Marriagerev −.45∗∗∗ 63.1 (38.1) on a 5-point scale from 1) Strongly disagree to 5) Strongly
Military .41∗∗∗ 57.9 (31.8) agree. The religious belief scale had excellent reliability
(𝛼 = .95). Unfortunately, there was a significant amount
Police Authority .36∗∗∗ 52.0 (30.3)
of missing data (N = 90) for the religious belief scale –
Men in Feminismrev −.31∗∗∗ 61.9 (29.6) perhaps because it was the only scale that was administered
War .30∗∗∗ 55.1 (31.9) using a matrix responding format (this was done because
Capital Punishment .24∗∗∗ 67.4 (31.2) our intention was to administer the scales identically as they
Sexism for Men .24∗∗∗ 48.7 (29.4) have been administered in past research).
Campus Free Speechrev −.23∗∗∗ 46.3 (32.5)
Microaggressionsrev −.15∗∗∗ 58.6 (26.3) Science beliefs. We attempted to expand our science be-
lief questionnaire by adding additional items for which po-
litical liberals might be expected to have more anti-scientific
AOT-E. Participants were either administered the original stances. Specifically, we asked about the following (in addi-
or the revised AOT-E scale, as outlined in Table 7. Reliability tion to the items from Study 1; see Table 9): the heritabil-
is good for both scales (original: 𝛼 = .72; revised: 𝛼 = ity of human intelligence, the role of genetics in success,
.74), albeit not as strong as in Study 1. Participants reported “detox” therapies, and nuclear power. However, as is evi-
being more actively open-minded when asked about opinions dent from Table 9, the only anti-scientific stance that was
(revised scale; M = 65.5, SD = 16.5) than beliefs (original more common among political liberals was opposition to
scale; M = 56.7, SD = 17.2), t(537) = 6.07, SE = 1.45, p nuclear power. Nonetheless, unlike Study 1, many of the
< .001. Whereas 43% of the sample were at or below the issues (6 out of 13) did not significantly correlate with po-
scale midpoint when asked about beliefs, only 20.4% were litical ideology. At any rate, the full scale had acceptable
at or below the scale midpoint when asked about opinions. reliability (𝛼 = .72).
Thus, although only a minority indicated a resistance to
evidence in both conditions, this was more common when Political party. In additional to the political ideology ques-
asked about beliefs than opinions. This is what would be tions that were administered in Study 1, we also asked par-
expected if the conflation of beliefs with religious beliefs was ticipants to indicate which political party they most strongly
causing some individuals to indicate a resistance to evidence. affiliate with: Democrat, Republican, Independent, Other.
Alternatively (or in addition), it is possible that people are The sample was fairly politically balanced: 37% Democrat,
simply more open to changing opinions (which may be issues 31% Republican, 29% Independent, and 3% “other”. We
of taste/preference) than beliefs (which may refer more to also asked them who they voted for in the 2016 Presidential
people’s position on issues of apparent fact). At any rate, the Election, about favorability toward Donald Trump, and to
revised AOT-E removed the apparent bias against religious indicate the likelihood that they would vote for Trump in
individuals (Stanovich & Toplak, 2019). the 2020 Presidential Election. These measures, along with
social and economic political ideology, were included in the
demographics section of the survey.
Cognitive Reflection Test (CRT). We used a re-worded
version (Pennycook & Rand, 2019b) of the three-item CRT
(Frederick, 2005). The CRT consists of words problems Demographics. Participants were given a demographic
that cue an incorrect intuitive response and that therefore questionnaire that included the following items: age, gen-
partially index one’s disposition to engage in reflective rea- der, English proficiency, education, income, and ethnicity.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 486

Table 9: Science beliefs scale, Study 2. Items are labelled with (rev) if they were reverse scored so that a higher score
indicates a more pro-scientific belief. Conservatism is the mean of the social and economic conservative political ideology
questions. Mean scores range from 0-100, with 0 indicating complete disagreement with the statement and 100 indicating
complete agreement.

Name Item Correlation (r) Mean (SD)

with conservatism
Global warming Global warming is at least partly caused by human activity and is a −.37∗∗∗ 73.8 (30.4)
serious problem for the environment.
Big Bang The big bang theory is, generally speaking, the best explanation we −.29∗∗∗ 56.4 (32.8)
have so far for the origin of our universe.
Evolution Evolution is the best explanation so far for our origins. −.27∗∗∗ 57.6 (35.2)
Old Earth The universe is billions of years old. −.24∗∗∗ 78.2 (28.5)
Stem Cells Stem cell research is a productive enterprise. −.19∗∗∗ 72.5 (25.1)
Vaccines/Autism Vaccines can cause autism in children. (rev) −.11∗ 34.6 (30.7)
Modern Medicine Modern medicine is the most effective means of treating most disease. −.06 70.6 (26.9)
Genetics An individual’s genes play an important role in their life success. −.06 56.3 (27.1)
Detoxing It is possible (and advisable) to "detox” the body from chemicals. (rev) −.05 68.6 (27.1)
IQ Heritability Human intelligence is moderately heritable (that is, intelligence is −.05 63.6 (26.5)
partly determined by genetics).
Tech Problems Technology causes more problems than it helps solve. (rev) −.04 39.1 (30.5)
GMO/Health Genetically modified foods are hazardous to human health. (rev) −.01 61.6 (29.7)
Nuclear Power Nuclear power is a safe and viable source of energy. .09∗ 49.6 (29.0)
***indicates p < .001, *indicates p < .05.

3.1.3 Procedure and conservative opinions (r’s = −.55 and −.36 for original
and revised, respectively). Nonetheless, as mentioned, the
Participants either completed the AOT-E at the beginning of revised AOT-E was a significant predictor in every case –
the survey or at the end (but before CRT and demographics). and, based on the norms from Gignac and Szodorai (2016),
The presentation order did not change the pattern of results most of the correlations were medium (r = .20) to large (r
and the aggregate results will therefore be reported. Other- = .30). Moreover, both AOT-E scales were generally more
wise, the order of the following measures was randomized strongly correlated with the measures of interest than was
for each participant (unlike Study 1, which used a fixed or- CRT performance. Indeed, every measure was significantly
der): 1) conspiracist ideation, 2) paranormal belief, 3) moral correlated with the revised AOT-E after controlling for CRT
values, 4) science beliefs, 5) political opinions and free mar- performance (all rpartial ’s > .16, all p’s < .015). Thus, it
ket ideology, and 6) religious belief and God skepticism. appears that one’s mere stance toward revising beliefs ac-
This block of questionnaires was followed by the CRT and, cording to evidence may play a role in what they believe (as
finally, demographics. adults) – a conclusion that is plainly supportive of the idea
that reasoning is largely effective (for some).
3.2 Results and Discussion The pattern of correlations for the individual political
opinion items was similar to Study 1 (albeit with slightly
As is evident from Table 10 – and again supportive of the weaker effect sizes; see Table 11). Both versions of the
“reasoning helps” perspective – both versions of the AOT-E AOT-E scale were significantly associated with liberal po-
scale were significantly associated with every other primary litical stances on almost every issue, with two exceptions.
measure. However, consistent with Stanovich and Toplak The first exception, as in Study 1, was that AOT-E did not
(2019), the correlation between the original AOT-E and re- correlate with believing that microaggressions are problem-
ligious belief (r = .42) was more than double the size of the atic or unproblematic (in Table 11 this is coded such that
correlation for the revised AOT-E (r = .20). The revised scale a higher score indicates believing that microaggressions are
also had decreased correlations with traditional moral values unproblematic). The only notable difference between the two
(r’s = −.37 and −.17 for original and revised, respectively) versions of the AOT-E (apart from the fact that the correla-

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 487

Table 10: Correlations (Pearson r) among primary measures in Study 2 (Lucid). Cronbach’s Alpha for each scale is listed
in brackets along the major diagonal. AOT-E = Actively Open-minded Thinking about Evidence. CRT = Cognitive Reflection
Test. MV = Moral Values. N = 270 for AOT-E (original). N = 268 for AOT-E (revised). N = 538 for intercorrelations (columns
3-14). N = 220, 229, and 448 for religious belief correlations (columns 1, 2, and 3-14, respectively).

1 2 3 4 5 6 7 8 9 10 11 12 13
1. AOT-E (original) (.72)
2. AOT-E (revised) (.74)
3. CRT .27∗∗∗ .19∗∗ (.64)
4. Conspiracy Beliefs −.20∗∗ −.25∗∗∗ −.13∗∗ (.93)
5. Paranormal Beliefs −.14∗ −.22∗∗∗ −.19∗∗∗ .50∗∗∗ (.95)
6. Traditional MV −.37∗∗∗ −.17∗∗ −.14∗∗ .14∗∗ .08 (.77)
7. Care/Fairness MV .12∗ .48∗∗∗ .06 .03 −.01 .48∗∗∗ (.88)
8. Social Conservatism ∗∗∗ ∗∗∗ ∗
−.33 −.26 −.10 −.02 −.10 ∗ .32∗∗∗ −.15∗∗
9. Fiscal Conservatism −.27∗∗∗ −.17∗∗ −.02 −.08 −.14∗∗ .28∗∗∗ −.10∗ .83∗∗∗
10. Free Market Ideology −.27∗∗∗ −.23∗∗∗ −.06 −.05 −.06 .21∗∗∗ −.16∗∗∗ .37∗∗∗ .39∗∗∗ (.60)
11. Conservative Opinions −.55∗∗∗ −.36∗∗∗ −.12∗∗ .07 −.02 .43∗∗∗ −.17∗∗∗ .62∗∗∗ .55∗∗∗ .47∗∗∗ (.72)
12. Pro-Science Beliefs .40∗∗∗ .39∗∗∗ .26∗∗∗ −.28∗∗∗ −.07 −.07 .33∗∗∗ −.32∗∗∗ −.21∗∗∗ −.20∗∗∗ −.40∗∗ (.72)
13. Religious Beliefs ∗∗∗ ∗∗∗
−.42 −.20 −.17 ∗∗∗ .22∗∗∗ .23∗∗∗ .49∗∗∗ .13∗∗ .32∗∗∗ .23∗∗∗ .22∗∗∗ .47∗∗∗ −.38∗∗∗ (.95)
14. God Skepticism .30∗∗∗ .26∗∗∗ .16∗∗∗ −.17∗∗∗ −.14∗∗ −.41∗∗∗ −.08 −.27∗∗∗ −.19∗∗∗ −.19∗∗∗ −.37∗∗∗ .31∗∗∗ −.67∗∗∗
***indicates p < .001, **indicates p < .01, *indicates p < .05.

tions tended to be stronger for the original than the revised The results for the science beliefs questionnaire largely
version) was that a more strongly pro-free speech stance was replicated Study 1 (Table 12). That is, every science be-
nominally negatively correlated (r = −.11, p = .070) with lief item that was included in both studies – including gen-
the original AOT-E, but significantly positively correlated (r eral trust in scientists – was positively correlated with both
= .19, p = .002) with the revised AOT-E; this correlation versions of the AOT-E (with the exception of the modern
was also positive using the original AOT-E in Study 1 (r = medicine item, which was only marginally correlated with
.25, p < .001; see Table 5). The latter correlation is notable the original AOT-E in Study 2, r = .12, p = .060). The re-
because, in the Study 2 Lucid sample, conservatives more sults for the new items that were added to Study 2 were more
strongly disagreed that “students should be able to block con- tepid. Although disbelief in the “detoxing the body of chem-
troversial speakers from giving talks at their university” – a icals” item was correlated with AOT-E, this was not true for
stance that was also associated with higher AOT-E (see also any of the other new items. If anything, having a positive
De keersmaecker, Bostyn, Hiel & Roets, 2020, for related stance on nuclear power (the only item positively correlated,
results); this correlation in the same direction in the Study however modestly, with political conservatism; see Table 9)
1 MTurk sample (−.08, Table 2) but was not significant. In was nominally (but not significantly) negatively associated
with the revised AOT-E (r = −.11, p = .087). Nonetheless,
other words, the campus free speech item is the only case
10 out of 13 items (along with general trust in scientists)
where higher AOT-E is associated with a stance (favoring
were correlated with the revised AOT-E in the expected di-
free speech) that is positively (although modestly) correlated
rection (see also; McPhetres & Pennycook, 2020). Thus,
with conservative political ideology (Table 8). All other is-
the results again support the contention that reasoning (on
sues were in the opposite direction (or non-significant, as is
balance) facilitates pro-science judgment.
the case for the microaggressions item).
Finally, as with the overall measures, CRT was a weaker
(and often non-significant) predictor for every item relative 4 Study 3
to either AOT-E scale. Combined with Study 1, these results
indicate that a major consequence of AOT-E is for political The results of Study 2 largely reinforced what we found in
ideology – precisely the domain where motivated reasoning Study 1: Believing that beliefs (or opinions) should change
is purported to dominate (but for a more direct test, see Study according to evidence was associated with skepticism about
3). conspiratorial, paranormal, and religious claims. Consistent

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 488

Table 11: Correlations (Pearson r) between AOT-E and political opinion items in Study 2 (Lucid). Opinion items are scored
such that a higher score corresponds with a more strongly politically conservative position and are organized in order of the
strength of positive association with conservatism (see Table 8). AOT-E = Actively Open-minded Thinking about Evidence.
CRT = Cognitive Reflection Test. N = 270 for AOT-E (original). N = 268 for AOT-E (revised). N = 538 for intercorrelations
(columns 3–14).

1 2 3 4 5 6 7 8 9 10 11 12 13
1. AOT-E (original)
2. AOT-E (revised)
3. CRT .27∗∗∗ .19∗∗
4. Abortion −.37∗∗∗ −.18∗∗ −.12∗∗
5. Same Sex Marriage −.37∗∗∗ −.33∗∗∗ −.14∗∗ .55∗∗∗
6. Military −.35∗∗∗ −.24∗∗∗ −.10∗ .25∗∗∗ .25∗∗∗
7. Police Authority −.29∗∗∗ −.32∗∗∗ −.06 .18∗∗∗ .22∗∗∗ .45∗∗∗
8. Men in Feminism ∗∗∗ ∗∗∗
−.35 −.32 −.16 ∗∗∗ .37∗∗∗ .46∗∗∗ .24∗∗∗ .16∗∗∗
9. War −.37∗∗∗ −.30∗∗∗ −.05 .13∗∗ .16∗∗∗ .48∗∗∗ .36∗∗∗ .19∗∗∗
10. Capital Punishment ∗ ∗∗
−.12 −.17 −.01 .08 .08 .42∗∗∗ .43∗∗∗ .11∗∗ .39∗∗∗
11. Sexism for Men ∗∗∗ ∗∗∗
−.32 −.21 −.05 .13∗∗ .12∗∗ .30∗∗∗ .25∗∗∗ .11∗ .30∗∗∗ .19∗∗∗
12. Campus Free Speech −.11 .19∗∗ .10∗ .25∗∗∗ .17∗∗∗ .13∗∗ <.01 .12∗∗ .05 .07 .07
13. Microaggressions −.09 −.08 −.04 .14∗∗ .11∗ .01 −.07 .16∗∗∗ .03 −.01 −.04 .15∗∗∗
14. Trust in Government −.21∗∗∗ −.24∗∗∗ −.08 −.09∗ −.04 .06 .22∗∗∗ .05 .19∗∗∗ .04 .19∗∗∗ −.18∗∗∗ −.08
***indicates p < .001, **indicates p < .01, *indicates p < .05.

with Stanovich and Toplak (2019), asking about opinions (re- gregate, it remains unclear if AOT-E is predictive of (in
vised AOT-E) instead of beliefs (original AOT-E) decreased particular) liberal opinions and pro-science beliefs across
(but did not wholly undermine) the correlation with religious the political spectrum. Indeed, previous research has shown
belief – nonetheless, the revised AOT-E continued to signifi- that cognitive sophistication interacts with political ideol-
cantly predict religious belief. Moreover, the revised AOT-E ogy when predicting people’s stance on issues such as global
was just as successful at predicting conspiratorial and para- warming (Kahan et al., 2012; Kahan, Peters, Dawson &
normal beliefs as the original AOT-E. Furthermore, as in Slovic, 2017; Sarathchandra, Navin, Largent & McCright,
Study 1, AOT-E was positively associated with care/fairness 2018). Unfortunately, because we assigned participants to
moral values and negatively associated with traditional moral two different AOT-E scales, we did not have enough power
values. Both versions of the AOT-E were also negatively in Study 2 to effectively estimate effect sizes when separat-
correlated with political conservatism; including political ing Democrats and Republicans.2 Given that the original
ideology, free market ideology, and a wide range of conser- “belief” version of the AOT-E may modestly inflate some
vative political opinions. The only exception was that the estimates of the correlation between AOT-E and a variety
revised AOT-E was positively associated with support for of issues (particularly those that have some association with
campus free speech. Although this item was only modestly religious belief), we ran a third study employing only the
associated with political conservatism (r = .09), it is note- revised “opinion” version of the AOT-E.
worthy that this is the sole issue out of the ten surveyed where
the more politically conservative stance was associated with 4.1 Method
the stance that beliefs should change according to evidence
(see also De keersmaecker et al., 2020). Both versions of 4.1.1 Participants
the AOT-E were also predictive of a number of pro-science
American participants were recruited from Lucid for Aca-
beliefs (with a few exceptions) (McPhetres & Pennycook,
demics on May 9th , 2019. We recruited 1000 participants.
2020). Overall, these results indicate that the AOT-E scale
In total, 1063 participants began the study but 103 did not
maintains strong predictive validity even if “opinions” are
referenced instead of “beliefs”. 2Only 97 Democrats, 88 Republicans, and 75 Independents were admin-
Although Study 2 paints a fairly clear picture in the ag- istered the revised AOT-E in Study 2.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 489

Table 12: Correlations (Pearson r) between AOT-E and science belief items in Study 2 (Lucid). Items are scored such that
a higher score corresponds with a more strongly pro-science stance and are organized in order of the strength and direction
of association with conservatism (see Table 9). AOT-E = Actively Open-minded Thinking about Evidence. CRT = Cognitive
Reflection Test. GMO = Genetically Modified Organism. N = 270 for AOT-E (original). N = 268 for AOT-E (revised). N = 538
for intercorrelations (columns 3–17).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1. AOT-E (original)
2. AOT-E (revised)
3. CRT .27∗∗∗ .19∗∗∗
4. Global warming .35∗∗∗ .28∗∗∗ .06
5. Big Bang .28∗∗∗ .20∗∗ .14∗∗ .34∗∗∗
6. Evolution .33∗∗∗ .17∗∗ .14∗∗ .28∗∗∗ .68∗∗∗
7. Old Earth .23∗∗∗ .25∗∗∗ .07 .31∗∗∗ .47∗∗∗ .46∗∗∗
8. Stem Cells .29∗∗∗ .33∗∗∗ .10∗ .42∗∗∗ .38∗∗∗ .40∗∗∗ .42∗∗∗
9. Vaccine/Autism .31∗∗∗ .38∗∗∗ .19∗∗∗ .21∗∗∗ .14∗∗ .14∗∗ .16∗∗∗ .28∗∗∗
10. Modern Med .12 .18∗∗∗ .08 .24∗∗∗ .27∗∗∗ .27∗∗∗ .28∗∗∗ .32∗∗∗ .27∗∗∗
11. Genetics .05 <.01 .04 .17∗∗∗ .21∗∗∗ .19∗∗∗ .15∗∗ .23∗∗∗ −.01 .21∗∗∗
12. Detoxing .14∗ .13∗ .12 −.13∗∗
∗∗ .04 ∗
.03 −.10 −.14 ∗∗ .24 −.07 −.13∗∗
∗∗∗

13. IQ Heritability .07 .11 .17∗∗∗ .15∗∗∗ .23∗∗∗ .18∗∗∗ .15∗∗ .31∗∗∗ .05 .30∗∗∗ .47∗∗∗ −.05
14. Tech Problems .20∗∗ .31∗∗∗ .12∗∗ .07 .05 .04 .07 .18∗∗∗ .37∗∗∗ .23∗∗∗ .02 .07 .03
15. GMO/Health .14∗ .14∗ .22∗∗∗ −.13∗∗ .08 .05 −.05 −.01 .30∗∗∗ .07 −.15 ∗∗ .42 −.10∗ .20∗∗∗
∗∗∗

16. Nuclear Power −.02 −.11 .15∗∗ −.05 .19∗∗∗ .13∗∗ .07 .19∗∗∗ .04 .24∗∗∗ .21∗∗∗ .03 .17∗∗∗ .03 .17∗∗∗
17. Trust Scientists .30∗∗∗ .27∗∗∗ .15∗∗ .41∗∗∗ .42∗∗∗ .43∗∗∗ .38∗∗∗ .47∗∗∗ .19∗∗∗ .34∗∗∗ ∗∗∗
.25 −.05 .28∗∗∗ .22∗∗∗ .03 .16∗∗∗
***indicates p < .001, **indicates p < .01, *indicates p < .05.

finish. We also removed 182 individuals who responded af- Table 13: Scale reliabilities (Cronbach’s Alpha) for Study
firmatively when asked if they responded randomly at any 3. AOT-E = Actively Open-minded Thinking about Evidence.
point during the survey. The resulting sample (N = 778, CRT = Cognitive Reflection Test. MV = Moral Values. N’s =
Mean age = 43.8) consisted of 363 males and 410 females, 302 Democrats, 223 Republicans, 239 Independents.
2 transgender males, 2 trans/non-binary, and 1 “not listed”.
Demo- Repub- Indepen-
crats licans dents
4.1.2 Materials and Procedure
AOT-E (revised) .72 .71 .73
The materials and procedure were identical to Study 2, with CRT .69 .73 .64
the following exceptions: 1) Participants were administered Conspiracy Beliefs .94 .94 .93
only the revised (“opinion”) AOT-E; 2) the religious be- Paranormal Beliefs .96 .95 .94
lief questionnaire (for which there was substantial missing Traditional MV .75 .71 .74
data in Study 2) was changed from a matrix presentation Care/Fairness MV .89 .85 .89
format to the single-question format used for other mea- Free Market Ideology .53 .64 .62
sures; 3) we also changed the response options for the reli-
Conservative Opinions .69 .52 .57
gious belief questionnaire to be consistent with the paranor-
Pro-Science Beliefs .68 .58 .65
mal/political/science questionnaires (i.e., a 7-point scale); 4)
Religious Belief .95 .92 .95
we added 3 CRT items from Thomson and Oppenheimer
(2016) that are relatively easier, based on past research (see
https://osf.io/xqzse/ for full materials); and 5) we added a sin-
Lean Republican, Republican, Strongly Republican), in ad-
gle continuous measure of Democrat-Republican preference
dition to the party classification item used in Study 2 (which
(“Which of the following best describes your political prefer-
included “independent” as an option).
ence?” Strongly Democratic, Democratic, Lean Democratic,

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 490

Table 14: Correlations (Pearson r) between AOT-E/CRT and primary measures in Study 3 (Lucid), separating Democrats
(N = 302), Republicans (N = 223), and Independents (N = 239). Overall N = 778. 3 AOT-E = Actively Open-minded Thinking
about Evidence. CRT = Cognitive Reflection Test. MV = Moral Values.

Con- Para- Tradi- Care/- Social Fiscal Free Conserv Pro- Religious God
spiracy normal tional Fairness Con- Con- Market Opin- Science Beliefs Skepti-
Beliefs Beliefs MV MV serv serv Ideol. ions Beliefs cism
AOT-E Dem −.32∗∗∗ −.33∗∗∗ −.26∗∗∗ .47∗∗∗ −.30∗∗∗ −.16∗∗ −.32∗∗∗ −.50∗∗∗ .46∗∗∗ −.27∗∗∗ .20∗∗∗
Rep −.12 −.14∗ .04 .35∗∗∗ .08 .14∗ .06 .05 .19∗∗ −.07 .02
Ind −.06 −.15∗ −.14∗ .34∗∗∗ −.12 .01 −.09 −.20∗∗ .32∗∗∗ −.16∗ .11
Overall −.19∗∗∗ −.22∗∗∗ −.16∗∗∗ .39∗∗∗ −.17∗∗∗ −.07 −.16∗∗∗ −.29∗∗∗ .35∗∗∗ −.20∗∗∗ .13∗∗∗
CRT Dem −.25∗∗∗ −.24∗∗∗ −.29∗∗∗ .10 −.31∗∗∗ −.18∗∗ −.21∗∗∗ −.39∗∗∗ .27∗∗∗ −.30∗∗∗ .25∗∗∗
Rep −.03 −.16∗ −.07 −.11 .19∗∗ .15∗ .20∗∗ .07 .10 <.01 .03
Ind −.11 −.18∗∗ −.13 .08 .05 .07 −.06 −.12 .21∗∗ −.16∗ .10
Overall −.15∗∗∗ −.21∗∗∗ −.17∗∗∗ .04 −.04 <.01 −.04 −.16∗∗∗ .18∗∗∗ −.17∗∗∗ .13∗∗∗
***indicates p < .001, **indicates p < .01, *indicates p < .05.

Scale reliabilities for Democrats, Republicans, and Inde- Furthermore, as in Study 2, the CRT results tended to parallel
pendents are in Table 13. Reliability was low for the free AOT-E despite being a weaker predictor overall.
market ideology scale and (among Republicans and Indepen- Turning now to specific political issues that formed our
dents in particular) the conservative opinion and pro-science conservative opinions scale (Table 15), it is evident that the
belief scales. Notably, variability was fairly similar across previously identified correlation between AOT-E and liberal
the major variables for Democrats and Republicans, indicat- political opinions (with one notable exception) was driven
ing that restriction of range is not a likely explanation for any largely by Democrats and, in some cases, by Independents.
divergences that we observe. For example, those higher in AOT-E were supportive of
same-sex marriage among Democrats (r = −.31) and In-
4.2 Results and Discussion dependents (r = −.27), but this correlation is marginally
significant among Republicans (r = −.12, p = .075). Most
Our focus for Study 3 was on the extent to which AOT- importantly, however, there was only a single issue where
E predicted the same constellation of beliefs, values, and AOT-E predicted opposite opinions for Democrats and Re-
opinion for individuals across the political spectrum.4 As publicans: Capital punishment. Whereas higher AOT-E
is evident from Table 14, AOT-E was a strong predictor was associated with more opposition to capital punishment
across the board for Democrats (paralleling the overall results among Democrats, it was associated with more support for
from Studies 1 and 2). However, the same was not equally capital punishment among Republicans. Thus, even though
true for Republicans, for whom AOT-E was a significant AOT-E was not as strongly predictive among Republicans as
predictor of skepticism about paranormal claims, acceptance Democrats, it is clear that it is not merely driving political
of care/fairness as moral values, and (notably) pro-science polarization either (lest more issues would be significantly
beliefs, but no other issues. Interestingly, in contrast with correlated in opposite directions). Indeed, the opinion that
the overall results, AOT-E was positively associated with there is room for men in feminism was associated with higher
economic conservatism among Republicans. Nonetheless, AOT-E for both Democrats and Republicans. Finally, the
it is noteworthy that Republicans scored themselves lower on previously noted exceptional case where the more conser-
AOT-E (M = 61.2, SD = 15) than both Democrats (M = 65.1, vative opinion was, overall, associated with higher AOT-E
SD = 15.9) and Independents (M = 64.9, SD = 16.3), t’s > – support for free speech – was driven by Republicans and
2.5, p’s < .015. Moreover, the correlations when considering Independents (i.e., AOT-E was not associated with support
the full sample (averaging across liberals and conservatives) for free speech among Democrats). Thus, whether AOT-E
paralleled the previous two studies: AOT-E was a significant predicts support or opposition for a specific issue appears to
predictor for every measure except economic conservatism. depend to some extent on what the issue is and about whom
4We will focus here on party affiliation, but the results are similar if lib- you’re speaking. Nonetheless, AOT-E certainly maintained
erals and conservatives are separated using the political ideology measures a great deal of predictive validity (contrary to the “reasoning
or the continuous party identification measure.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 491

Table 15: Correlations (Pearson r) between AOT-E/CRT and political opinion measures in Study 3 (Lucid), separating
Democrats (N = 302), Republicans (N = 223), and Independents (N = 239). Opinion items are scored such that a higher
score corresponds with a more strongly politically conservative position and are organized in order of the strength of positive
association with conservatism. AOT-E = Actively Open-minded Thinking about Evidence. CRT = Cognitive Reflection Test.

Abortion Same Military Capital Men in Police ..War.. Free Men Micro- Gov’t
Sex Punish- Femi- Author- Speech Sexism aggres- Trust
Mar- ment nism ity sions
riage

AOT-E Dem −.23∗∗∗ −.31∗∗∗ −.37∗∗∗ −.28∗∗∗ −.34∗∗∗ −.37∗∗∗ −.29∗∗∗ .10 −.37∗∗∗ −.11 −.37∗∗∗
(revised) Rep .01 −.12 .08 .21∗∗ −.14∗ −.04 −.06 .26∗∗∗ .02 −.01 −.18∗∗
Ind −.09 −.27∗∗∗ −.06 −.03 −.20∗∗ −.18∗∗ −.15∗ .24∗∗∗ −.12 −.03 −.36∗∗∗
CRT Dem −.25∗∗∗ −.23∗∗∗ −.31∗∗∗ −.30∗∗∗ −.17∗∗ −.21∗∗∗ −.29∗∗∗ .06 −.22∗∗∗ −.06 −.20∗∗∗
Rep .04 .05 −.11 −.01 −.05 −.04 .05 .18∗∗ .09 .06 −.10
Ind −.14∗ −.07 .02 .05 −.09 −.08 −.05 .05 −.13∗ −.11 −.09
***indicates p < .001, **indicates p < .01, *indicates p < .05.

Table 16: Correlations (Pearson r) between AOT-E/CRT and pro-science belief measures in Study 3 (Lucid), separating
Democrats (N = 302), Republicans (N = 223), and Independents (N = 239). Items are scored such that a higher score
corresponds with a more strongly pro-science stance and are organized in order of the strength and direction of association
with conservatism (see Table 17). AOT-E = Actively Open-minded Thinking about Evidence. CRT = Cognitive Reflection Test.
GMO = Genetically Modified Organisms.

Global Big Evolu- Old Stem Vaccine/- Tech Modern IQ Detox- Gene- GMO/- Nuclear Scien-
Warm- Bang tion Earth Cells Autism Prob- Med Heri- ing tics Health Power tist
ing lems table Trust

AOT-E Dem .43∗∗∗ .24∗∗∗ .28∗∗∗ .27∗∗∗ .38∗∗∗ .47∗∗∗ .39∗∗∗ .19∗∗ .16∗∗ .11 −.05 .15∗∗ −.20∗∗ .29∗∗∗
(revised) Rep .09 <.01 −.07 .05 .28∗∗∗ .19∗∗ .24∗∗∗ .14∗ .12 −.02 .12 .03 −.06 .14∗
Ind .25∗∗∗ .15∗ .22∗∗ .24∗∗∗ .26∗∗∗ .18∗∗ .23∗∗∗ .13∗ .24∗∗∗ −.13∗ .08 −.03 −.03 .18∗∗

CRT Dem .19∗∗ .13∗ .22∗∗∗ .09 .16∗∗ .30∗∗∗ .18∗∗ .02 −.02 .16∗∗ −.06 .17∗∗ −.01 .20∗∗
Rep −.12 −.02 −.04 −.10 .10 .13 .01 .05 .06 <.01 .08 .21∗∗ .19∗∗ −.04
Ind .12 .03 .11 .14∗ .16∗ .24∗∗∗ .04 .11 .07 .01 −.03 .18∗∗ .02 .19∗∗
***indicates p < .001, **indicates p < .01, *indicates p < .05.

is helpless” perspective and consistent with the “reasoning politically polarizing issues, such as global warming5, big
is helpful” perspective) and was not associated with politi- bang, and evolution (see Table 17), were not associated with
cal polarization writ large (underming the “reasoning hurts” AOT-E. However, AOT-E was associated with pro-science
perspective). stances on several intermediate issues; namely support for
The results for science-related beliefs (Table 16) parallel stem cells, vaccines, technology, and modern medicine. Fur-
the pattern for conservative opinions insofar as they provide thermore, trust in scientists was positively associated with
evidence against the motivated reasoning (“reasoning hurts”) AOT-E across the political spectrum.
perspective. Specifically, AOT-E was generally associated
with pro-science beliefs across the board for Democrats (with
5Consistent with Kahan and Corbin (2016), we found a significant in-
the exceptions of skepticism about detoxing and the role of teraction between belief in anthropogenic global warming and political
genetics in success, which were not significant, and sup- partisanship (Democrat vs. Republican) in predicting AOT-E, 𝛽 = .30, p
porting nuclear power, which was negatively associated with < .001. However, as shown in Table 16, the coefficient for the correlation
between global warming and AOT-E among Republicans is actually positive
AOT-E) and Independents (with the exceptions of genetics (instead of negative), which is contrary to the claim that AOT-E is associated
and supporting GMO’s). Among Republicans, the most with more skepticism about global warming among Republicans.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 492

Table 17: Science beliefs scale correlations with conser- which motivated reasoning should be prominent); from hav-
vatism, Study 3. Science beliefs have been re-scaled such ing a more liberal political ideology, to adoption of more
that a higher value indicates a more pro-science belief. Con- liberal moral values (specifically, rejection of traditional val-
servatism is the mean of the social and economic conserva- ues and agreement with care/fairness values), to opposition
tive political ideology questions. Mean scores range from 0- to economic conservatism and free-market ideology. Fur-
100, with 0 indicating complete disagreement with the state- thermore, AOT-E was positively associated with a variety of
ment and 100 indicating complete agreement. specific liberal political opinions (e.g., supporting gay mar-
riage and access to abortion) in the aggregate. This suggests
Name Correlation (r) Mean (SD) that political conservatives, who tend to be more resistant
with conservatism to societal change (White, Kinney, Danek, Smith & Harben,
2019), may also be more resistant to intrapersonal belief
Global warming −.34∗∗∗ 74.3 (28.1) change (but see below for a more nuanced interpretation).
Big Bang −.29∗∗∗ 57.5 (31.8) The strong predictive validity of the AOT-E across a wide
Evolution −.25∗∗∗ 60.1 (33.1) range of domains suggests that people’s meta-beliefs about
whether and how beliefs should change play an important
Old Earth −.18∗∗∗ 77.5 (26.7)
role in belief formation. However, this conclusion comes
Stem Cells −.16∗∗∗ 71.8 (23.4) with an important caveat that is in many ways as interest-
Vaccines/Autism −.05 65.7 (31.3) ing as the conclusion itself. Most notably, Study 3 revealed
Tech Problems −.03 55.5 (29.7) that AOT-E is much more predictive among Democrats than
Modern Medicine −.02 69.1 (26.6) among Republicans (with Independents being intermedi-
ate).6 This interaction is, in some cases, consistent with
IQ Heritability −.003 66.6 (23.9) previous research that has been used to support the “reason-
Detoxing .01 31.2 (25.6) ing hurts” perspective – however, as we will argue, it is not
Genetics .02 60.4 (26.2) consistent with how some of these past results have been
GMO/Health .03 37.9 (28.2) interpreted.
Nuclear Power .12∗∗ 48.5 (28.9)
***indicates p < .001, **indicates p < .01. 5.1 AOT-E among Democrats and Republi-
cans
The AOT-E did not have the same predictive validity for con-
5 General Discussion servatives as it did for liberals. To take a prominent example
from Study 3, AOT-E was very strongly correlated with be-
Although the belief that beliefs (and opinions) ought to lief in anthropogenic climate change among Democrats (r =
change according to evidence is held by most people, there is .43, p < .001), but there was no such (significant) correla-
meaningful variability in the strength of this conviction. The tion among Republicans (r = .09, p = .179). This parallels
results of all three studies point to one broad, yet important, previous findings wherein individuals who are more cogni-
conclusion: Actively open-minded thinking about evidence tively sophisticated (using a variety of measures, including
(AOT-E) is, in the aggregate, a strong predictor of a wide the CRT) are more politically polarized in terms of climate
range of beliefs, opinions, and values. This implies that change (Kahan et al., 2012). In particular, Kahan et al. found
individual differences in the propensity to reflect about evi- that science literacy and numeracy was positively correlated
dence is something that people meaningfully engage in their with climate change risk attitudes among liberals (r = .08,
everyday lives, which indicates that the exercise of human p = .03) but negatively correlated among conservatives (r =
reasoning is, on balance, “helpful”. The respective ideas that −.12, p = .03).7 The favored explanation for these results is
reasoning is “helpless” or “hurtful” did not find support in that cognitive sophistication polarizes climate change (and
our data.
To summarize, AOT-E was associated with skepticism 6Nonetheless, AOT-E was significantly correlated with every primary
about conspiratorial, paranormal, and religious claims and measure in all three studies even after political ideology and demographics
were statistically controlled (see supplementary materials for full analysis).
agreement with a variety of scientific claims. Thus, AOT-
The same was true for CRT performance (albeit to a lesser extent), except
E appears to support the rejection of epistemically suspect for care/fairness moral values and free market ideology.
beliefs, thereby indicating that what people believe about 7We see a more direct parallel when considering the association between
whether beliefs ought to change (meta-beliefs) has an influ- CRT and belief in anthropogenic global warming in Study 3: r = .19, p =
.001 among Democrats; r = −.12, p = .088 among Republicans. Further-
ence on what they take to be true or false about the world.
more, Kahan and Corbin (2016) also found a significant positive correlation
AOT-E was also consistently associated with political lib- between views on climate change and AOT among Democrats and a null
eralism in a variety of forms (despite being a domain for (but nominally negative) correlation among Republicans.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 493

other) attitudes because it facilitates motivated (“identity- CRT in our own data (with the former being a stronger pre-
protective”) reasoning (Kahan et al., 2012; Kahan, Peters, dictor overall than the latter) as well as the data of Kahan
Dawson & Slovic, 2017; Sarathchandra, Navin, Largent & and Corbin (Baron, 2017). Indeed, as intimated in the in-
McCright, 2018) – an account that has notably been applied troduction, AOT-E is definitionally opposed to motivated
to the AOT as a measure of general cognitive sophistication reasoning: Believing that beliefs ought to change according
as well (Kahan & Corbin, 2016; but see Baron, 2017). This to evidence essentially amounts to a rejection of motivated
“humans-as-lawyers” motivated reasoning perspective has reasoning. Of course, it is possible that those who report be-
had a large influence on the field and in the popular press ing more actively open-minded are simply being deceptive
(for an overview, see Pennycook, 2018). (and potentially self-deceptive). That is, individuals who
Motivated reasoning cannot, however, account for our say that they are particularly willing to change their beliefs
broad pattern of results (with some potential exceptions). according to evidence are, in reality, the most likely to do
Although an interaction between political ideology and cog- the opposite and engage in motivated reasoning. This seems
nitive sophistication in the prediction of an attitude (such as unlikely. And, at any rate, the results for the CRT – which
climate change belief) is often taken as positive evidence for is plainly a measure of some sort of cognitive sophistica-
the motivated reasoning account, the interaction is easily un- tion – also do not support the motivated reasoning account.
derstood as a consequence of the sample characteristics. For There was not a single specific issue, political or science-
example, the interaction between political ideology and cog- based, that produced opposing and significant correlations
nitive sophistication in the prediction of climate change atti- with CRT for Democrats relative to Republicans. Thus, the
tudes emerges because the sample happens to have both lib- most parsimonious take-away from the present data is sim-
erals and conservatives. However, an interaction can emerge ply that the motivated reasoning account (a “reasoning hurts”
from opposing effects that are not individually significant. perspective) is wrong or incomplete.
This is important because the central prediction of the moti- If not motivated reasoning, what then explains the finding
vated reasoning (“reasoning hurts”) account is actually two that AOT-E is more consistently predictive for liberals than
separate (and opposing) main effects for political liberals and conservatives? It is potentially revealing that the depression
conservatives: Cognitive sophistication should be positively in predictive validity for Republicans relative to Democrats
associated with politically congruent attitudes (e.g., climate was seemingly evident even for measures that did not signifi-
change for liberals) and negatively associated with politically cantly correlate with conservatism. For example, conspiracy
incongruent attitudes (e.g., climate change for conservatives; ideation was strongly correlated with AOT-E for Democrats
see Pennycook & Rand, 2019b). The results of Study 3 are (r = −.32, p < .001) but less so for Republicans (although
plainly inconsistent with this prediction: Not only did we not it was marginally significant, r = −.12, p = .070).8 This
find opposing main effects in the context of climate change occurred even though conspiracy belief was equivalent be-
(in fact, AOT-E is nominally positively correlated with cli- tween Democrats and Republicans (t = 1.16, p = .245). One
mate change beliefs among Republicans), we do not find mundane possibility is that data quality was, for whatever
it for any other polarized issue either (with one exception). reason, poorer among Republicans than Democrats. Con-
Specifically, there was not a single scientific issue that we trary to this, scale reliabilities were largely similar for both
included in our study that produced significant correlations groups (Table 13). Moreover, random responding was very
with AOT-E in opposite directions for Democrats and Re- similar for Republicans and Democrats.
publicans. Furthermore, across ten explicitly political issues One possibility is that there are important differences be-
(such as support for police authority or opposition to abor- tween the “coalitions” that make up the Democratic and Re-
tion) there was only one case where the motivated reasoning publican parties. For example, Baron (2017) noted that the
prediction of significant opposing effects was present: Capi- Democratic Party in the United States (and liberals in gen-
tal punishment. Even broad ideological positions such as so- eral) is made up of a more diverse group of people than is the
cial conservatism and free-market ideology did not produce Republican Party. Supporting this idea, variation in most of
significant opposing effects (although a second exception the primary measures in Studies 2 and 3 (including the AOT-
is present here: Economic conservatism). Thus, in almost E itself) is at least nominally higher among Democrats than
every case, the motivated reasoning (or identity-protective Republicans (see Table S4 in the supplementary materials).
cognition) account’s prediction was not supported. Rather, Yet another possibility is that there is no genuine differ-
it appears that AOT-E is simply less predictive among polit- ence between conservatives and liberals in terms of people’s
ical conservatives than among liberals. beliefs about how beliefs should change, but the AOT-E items
One potential explanation of this is that, despite argu- are viewed through a political lens in the similar sort of way
ments to the contrary (Kahan & Corbin, 2016), actively that the “belief” items in the original AOT-E were biased
open-minded thinking about evidence is not merely a proxy
for cognitive sophistication (Baron et al., 2015) – an ob- 8However, the interaction between AOT-E and a continuous measure
of political partisanship (Democrat vs. Republican) in the prediction of
servation supported by the divergences between AOT-E and conspiracy beliefs was only marginally significant, 𝛽 = .06, p = .093.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 494

against religious individuals (Stanovich & Toplak, 2019). counter-example to Funder and Ozer, Study 1 consisted of
It may be that “evidence” as a term or concept has been a large sample (N = 375) and was largely replicated with
politicized to some extent and that AOT-E may be viewed a different sample (using the original AOT-E) in Study 2
as a liberal outlook (Krugman, 2019). Under this account, (r’s were .55 and .40 for liberal opinions and pro-science
the weaker correlations among Republicans occurs because beliefs, respectively). Nonetheless, consistent with Funder
some proportion of conservatives are reporting lower AOT- and Ozer’s larger point, the very large effect sizes in Study 1
E simply because they are resisting the framing or wording may be inflated for two reasons.
of the questions (or perhaps the source of the questions – First, as argued by Stanovich and Toplak (2019) (who
for more on insincere responding in the context of partisan also noted the large effect sizes as a reason for skepticism),
bias in surveys, see Bullock & Lenz, 2019). One counter to the original version of the AOT-E appears to have inflated
this possibility, however, is that the CRT is also less predic- some effect sizes because individuals may have presumed
tive among Republicans. Republicans did no worse on the the questions to be about religious belief in particular in-
test than Democrats and presumably are not ideologically stead of beliefs more generally. Although religious believers
opposed to simple-seeming word problems. Although this continue to rate themselves as more resistant to revising
does not rule out the politicization of evidence possibility, it opinions according to evidence relative to non-believers, the
does render it less likely. “belief” wording in the original AOT-E (which was derived
Alternatively, liberals and conservatives (in the USA) from earlier scales) may have inflated the correlation with
may genuinely differ not only in what they believe (includ- religious belief and its covariates. However, one alternative
ing meta-beliefs), but why they believe it. The AOT-E is possibility is that the belief items are simply more predictive
equipped to assess one’s stance toward evidence, which is overall (e.g., because they are more easily understood). A
apparently important among liberals (insofar as AOT-E dis- more systematic investigation of “belief” versus “opinion”
tinguishes between what types of beliefs and opinions lib- wording is necessary to come to firm conclusions. Of course,
erals tend to have – although, of course, other factors are the present data indicate that the AOT-E is relatively strongly
surely important as well). At least based on the present cor- predictive regardless of these small changes in wording.
relational data, belief formation appears to be driven more by Second, as discussed, we found that AOT-E was much
other factors for conservatives. That is, it is not simply that more predictive across the board for political liberals
conservatives are less willing to change their beliefs accord- (Democrats) than for conservatives (Republicans). This is
ing to evidence (although the overall difference is nonethe- relevant for the apparently over-estimated correlation effect
less evident), but rather that factors unstudied here contribute sizes in Study 1 because that sample came from Mechani-
more substantially to belief formation among conservatives. cal Turk, which was heavily liberal-skewed.9 Thus, although
One of the apparent defining features of conservatism, apart Study 1 consisted of a large sample and produced results that
from resistance to change, is the endorsement of hierarchies were replicated in Study 2, our evidence indicates that Fun-
(e.g., Jost, Glaser, Kruglanski & Sulloway, 2003). Perhaps der & Ozer’s (2019) conclusion about very large effect sizes
part of the reason why AOT-E is less predictive among con- being likely overestimates is nonetheless accurate. However,
servatives, then, is that beliefs are less intrapersonal and in this case, the issue was more a matter of generalizability
more interpersonal among political conservatives. That is, than replicability. The underlying lesson is the same: Very
belief is more about social groups and, therefore, variation large effect sizes should be interpreted with caution.
in beliefs among conservatives is driven more by exposure
to different hierarchies and information sources (for an ex-
5.3 Limitations
ample, see Landrum, Lull, Akin, Hasell & Jamieson, 2017).
Plainly, further research is required. The principal limitation of the present study is that it is corre-
lational and therefore not possible to establish, for example,
whether AOT-E affects political opinions, vice versa, and/or
5.2 Very large effect sizes: A lesson some third factor affects both. Nonetheless, experimentally
In a recent discussion of effect size estimates, Funder and manipulating AOT-E and testing for a change in beliefs does
Ozer (2019) argued that r’s of .10, .20, and .30 correspond not seem a prudent approach. Beliefs, opinions, and values
to small, medium, and large effect sizes, respectively (see are formed across years, and minute-long manipulations do
also Gignac & Szodorai, 2016). They also argued that very not offer a reasonable proxy for the psychological processes
large effect sizes (r = .40 or greater) are, in the context of 9It is also noteworthy that reliability for the original AOT-E was greater
psychological research, “likely to be a gross overestimates for the Mechanical Turk sample (.87) than the Lucid sample (.72). It is
rarely found in a large sample or in a replication” (p. 1). In possible that the muted effect sizes in Studies 2 and 3 are also the result of
lower data quality on Lucid than on Mechanical Turk (and, indeed, random
Study 1, AOT-E predicted multiple beliefs and opinions at a
responding was much more common in the former than the latter). This
level greater than .40 (including aggregate liberal opinions may also explain why the CRT was broadly less predictive in the Lucid
and pro-science beliefs at r’s = ˜.60). As a meaningful sample than in past studies. CRT scores were, overall, quite low on Lucid.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 495

that are of chief interest here. Rather, longitudinal studies cognitive processes. Trends in Cognitive Sciences, 16(12),
that establish differences in AOT-E at adolescence and test 593–605. https://doi.org/10.1016/j.tics.2012.10.002.
for changes in beliefs over time would be a more fruitful Baron, J. (1985). What kinds of intelligence components are
future direction. fundamental? In S. F. Chipman, J. W. Segal, & R. Glaser
The generalizability of this study is limited in a number (Eds.), Thinking and learning skills: Vol. II, Research
of ways. First, our samples are not precisely representative and open questions. Hillsdale, NJ: Lawrence Earlbaum.
of the United States population (although Lucid is much Baron, J. (2008). Thinking and deciding (4th ed.). Cam-
closer than Mechanical Turk). Second, we obviously cannot bridge: Cambridge University Press.
generalize beyond the USA. Third, although we attempted Baron, J. (2017). Comment on Kahan and Corbin: Can
to test as many different beliefs, values, and opinions as polarization increase with actively open-minded thinking?
possible in a single survey, we have surely missed many Research & Politics, 4(1), 1–4. https://doi.org/10.1177/
important issues. Furthermore, it is possible that our own 2053168016688122.
selection of issues was subject to our own liberal political Baron, J. (2019). Actively open-minded thinking in poli-
bias. tics. Cognition. https://doi.org/10.1016/j.cognition.2018.
10.004.
Baron, J., & Jost, J. T. (2019). False equivalence: Are
6 Conclusion liberals and conservatives in the United States equally
biased? Perspectives on Psychological Science, 14(2),
Our 8-item actively open-minded thinking about evidence 292–303. https://doi.org/10.1177/1745691618788876.
(AOT-E) scale was strongly predictive of a wide range of Baron, J., Scott, S., Fincher, K., & Emlen Metz, S. (2015).
beliefs, values, and opinions. People who reported believ- Why does the Cognitive Reflection Test (sometimes) pre-
ing that beliefs and opinions should change according to dict utilitarian moral judgment (and other things)? Jour-
evidence were less likely to be religious, less likely to hold nal of Applied Research in Memory and Cognition, 4(3),
paranormal and conspiratorial beliefs, more likely to believe 265–284. https://doi.org/10.1016/j.jarmac.2014.09.003.
in a variety of scientific claims, and were more political lib- Brotherton, R., & French, C. (2013). Measuring belief
eral (in terms of overall ideology, partisan affiliation, moral in conspiracy theories: The generic conspiracist beliefs
values, and a variety of specific political opinions). More- scale. Frontiers in Psychology. https://doi.org/10.3389/
over, the effect sizes for these correlations was often large fpsyg.2013.00279.
or very large, based on established norms (Funder & Ozer, Bullock, J. G., & Lenz, G. (2019). Partisan bias in sur-
2019; Gignac & Szodorai, 2016). The size and diversity veys. Annual Review of Political Science, 22(1), 325–342.
of AOT-E correlates strongly supports one major, if broad, https://doi.org/10.1146/annurev-polisci-051117-050904.
conclusion: Socio-cognitive theories of belief (both specific Campitelli, G., & Gerrans, P. (2014). Does the cognitive
and general) should take into account what people believe reflection test measure cognitive reflection? A mathemat-
about when and how beliefs and opinions should change (i.e., ical modeling approach. Memory & Cognition, 42(3),
meta-beliefs). That is, we should not assume that evidence 434–447. https://doi.org/10.3758/s13421-013-0367-9.
is equally important for everyone. Furthermore, our findings Cesario, J. (2014). Priming, replication, and the hardest sci-
provide clear support for the perspective that reasoning fa- ence. Perspectives on Psychological Science, 9(1), 40–48.
cilitates sound judgment, thereby undermining the idea that https://doi.org/10.1177/1745691613513470.
intuition commonly dominates reasoning. We also found Cohen, P., Cohen, J., Aiken, L. S., & West, S. G. (1999).
essentially no support for motivated reasoning. Regardless The problem of units and the circumstance for POMP.
future work is required to more clearly delineate why AOT-E Multivariate Behavioral Research, 34, 315–346. https://
is more predictive for political liberals than conservatives. doi.org/10.1207/S15327906MBR3403_2.
Coppock, A., & Mcclellan, O. A. (2019). Validating the
demographic, political, psychological, and experimental
References results obtained from a new source of online survey re-
Bargh, J. A. (1999). The cognitive monster: The case against spondents. Research & Politics. Retrieved from https://
the controllability of automatic stereotype effects. In S. alexandercoppock.com/papers/CM_lucid.pdf.
Chaiken & Y. Trope (Eds.), Dual-process theories in so- De keersmaecker, J., Bostyn, D. H., Hiel, A. Van, & Roets,
cial psychology (pp. 361–382). Guilford Press. A. (2020). Disliked but free to speak: Cognitive ability is
Bargh, J. A., & Chartrand, T. L. (1999). The unbearable related to supporting freedom of speech for groups across
automaticity of being. American Psychologist, 54(7), the ideological spectrum. PsyArXiv, 1–21. https://doi.
462–479. https://doi.org/10.1037/0003-066X.54.7.462. org/https://doi.org/10.31234/OSF.IO/B7KTY
Bargh, J. A., Schwader, K. L., Hailey, S. E., Dyer, R. De Neys, W. (2017). Dual process theory 2.0. London, UK:
L., & Boothby, E. J. (2012). Automaticity in social- Routledge.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 496

Dijksterhuis, A., & Strick, M. (2016). A case for think- Haran, U., Ritov, I., & Mellers, B. a. (2013). The role
ing without consciousness. Perspectives on Psycholog- of actively open-minded thinking in information acqui-
ical Science, 11(1), 117–132. https://doi.org/10.1177/ sition, accuracy, and calibration. Judgment and De-
1745691615615317. cision Making, 8(3), 188–201. https://doi.org/10.1017/
Ditto, P. H., Liu, B. S., Clark, C. J., Wojcik, S. P., Chen, E. CBO9781107415324.004.
E., Grady, R. H., & Zinger, J. F. (2019). At least bias is Heath, Y., & Gifford, R. (2006). Free-market ideology and
bipartisan: A meta-analytic comparison of partisan bias in environmental degradation the case of belief in global
liberals and conservatives. Perspectives on Psychological climate change. Environment and Behavior, 38, 48–71.
Science, 14, 273–291. https://doi.org/10.1007/s10551- https://doi.org/10.1177/0013916505277998.
015-2769-z.For. Jost, J. T. (2017). Ideological asymmetries and the essence
Evans, J. S. B. T., & Stanovich, K. E. (2013). Dual-process of political psychology. Political Psychology, 38(2),
theories of higher cognition: Advancing the debate. 167–208. https://doi.org/10.1111/pops.12407.
Perspectives on Psychological Science, 8(3), 223–241. Jost, J. T., Glaser, J., Kruglanski, A. W., & Sulloway, F. J.
https://doi.org/10.1177/1745691612460685. (2003). Political conservatism as motivated social cog-
Evans, J. St. B. T. (2008). Dual-processing accounts of nition. Psychological Bulletin, 129(3), 339–375. https://
reasoning, judgment, and social cognition. Annual Re- doi.org/10.1037/0033-2909.129.3.339.
view of Psychology, 59, 255–278. https://doi.org/10.1146/ Kahan, D. M. (2013). Ideology, motivated reasoning, and
annurev.psych.59.103006.093629. cognitive reflection. Judgment and Decision Making,
Evans, J. St. B. T. (2012). Dual process theories of deductive 8(4), 407–424. https://doi.org/10.2139/ssrn.2182588.
reasoning: facts and fallacies. In The Oxford handbook of Kahan, D. M., & Corbin, J. C. (2016). A note on the per-
thinking and reasoning (pp. 115–133). New York: Oxford verse effects of actively open-minded thinking on climate-
U. change polarization. Research and Politics, 1–5. https://
Frederick, S. (2005). Cognitive reflection and decision mak- doi.org/10.1177/2053168016676705.
ing. Journal of Economic Perspectives, 19(4), 25–42. Kahan, D. M., Peters, E., Wittlin, M., Slovic, P., Ouellette, L.
https://doi.org/10.1257/089533005775196732. L., Braman, D., & Mandel, G. (2012). The polarizing im-
Funder, D. C., & Ozer, D. J. (2019). Evaluating ef- pact of science literacy and numeracy on perceived climate
fect size in psychological research: Sense and non- change risks. Nature Climate Change, 2(10), 732–735.
sense. Advances in Methods and Practices in Psycholog- https://doi.org/10.1038/nclimate1547.
ical Science, 251524591984720. https://doi.org/10.1177/ Kahan, D., Peters, E., Dawson, E., & Slovic, P. (2017). Mo-
2515245919847202. tivated numeracy and enlightened self-government. Be-
Gervais, W. M. (2015). Override the controversy: Analytic havioural Public Policy, 1(1), 54–86.
thinking predicts endorsement of evolution. Cognition, Kahneman, D, Slovic, P., & Tversky, A. (1982). Judgments
142, 312–321. https://doi.org/10.1016/j.cognition.2015. under uncertainty: Heuristics and biases. Cambridge,
05.011. MA: Cambridge University Press.
Gigerenzer, G. (2007). Gut feelings: The intelligence of the Kahneman, Daniel, & Klein, G. (2009). Conditions for
unconscious. Penguin Books. intuitive expertise: a failure to disagree. The American
Gigerenzer, G., Todd, P., & Group, the A. R. (1999). Simple Psychologist, 64(6), 515–526. https://doi.org/10.1037/
heuristics that make us smart. Oxford, England: Oxford a0016755.
University Press. Klein, G. (2008). Naturalistic decision making. Hu-
Gignac, G. E., & Szodorai, E. T. (2016). Effect size guide- man Factors, 50(3), 456–460. https://doi.org/10.1518/
lines for individual differences researchers. Personality 001872008X288385.
and Individual Differences, 102, 74–78. https://doi.org/ Knobloch-Westerwick, S., Mothes, C., & Polavin, N.
10.1016/j.paid.2016.06.069. (2017). Confirmation bias, ingroup bias, and negativ-
Graham, J., Nosek, B. A., Haidt, J., Iyer, R., Koleva, S., & ity bias in selective exposure to political information.
Ditto, P. H. (2011). Mapping the moral domain. Journal Communication Research, 1–21. https://doi.org/10.1177/
of Personality and Social Psychology, 101(2), 366–385. 0093650217719596.
https://doi.org/10.1037/a0021847. Kohlberg, L. (1969). Stage and sequence: The cognitive-
Haidt, J. (2012). The righteous mind: Why good people are developmental approach to socialization. In D. A. Goslin
divided by politics and religion. New York, NY: Paragon. (Ed.), Handbook of socialization theory and research (pp.
Haidt, Jonathan. (2001). The emotional dog and its rational 347–480).
tail: A social intuitionist approach to moral judgment. Krugman, P. (2019, April). Republicans are
Psychological Review, 108(4), 814–834. https://doi.org/ the real extremists. The New York Times.
10.1037//0033-295X. Retrieved from https://www.nytimes.com/2019/04/15/
opinion/republicans-ocasio-cortez-omar.html.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 497

Kunda, Z. (1990). The case for motivated reasoning. Psy- predicts religious and paranormal belief. Cognition,
chological Bulletin, 108(3), 480–498. https://doi.org/10. 123(3), 335–346. https://doi.org/10.1016/j.cognition.
1037/0033-2909.108.3.480. 2012.03.003.
Landrum, A. R., Lull, R. B., Akin, H., Hasell, A., & Pennycook, G., Fugelsang, J. A., & Koehler, D. J. (2015a).
Jamieson, K. H. (2017). Processing the papal encycli- Everyday consequences of analytic thinking. Current
cal through perceptual filters: Pope Francis, identity- Directions in Psychological Science, 24(6), 425–432.
protective cognition, and climate change concern. Cogni- https://doi.org/10.1177/0963721415604610.
tion, 166, 1–12. https://doi.org/10.1016/j.cognition.2017. Pennycook, G., Fugelsang, J. A., & Koehler, D. J. (2015b).
05.015. What makes us think? A three-stage dual-process model
Lodge, M., & Taber, C. (2005). The automaticity of af- of analytic engagement. Cognitive Psychology, 80, 34–72.
fect for political candidates, parties and issues. Political https://doi.org/10.1016/j.cogpsych.2015.05.001.
Psychology, 26(3), 455–482. Pennycook, G., & Rand, D. G. (2019a). Cognitive reflection
McPhetres, J., & Pennycook, G. (2020). Science beliefs, and the 2016 US presidential election. Personality and
political ideology, and cognitive sophistication. PsyArXiv Social Psychology Bulletin, 45, 224–239. https://doi.org/
Working Paper, 1–33. https://doi.org/10.31219/OSF.IO/ 10.1177/0146167218783192.
AD9V7. Pennycook, G., & Rand, D. G. (2019b). Lazy, not biased:
Mercier, H. (2016). The argumentative theory: Predictions Susceptibility to partisan fake news is better explained by
and empirical evidence. Trends in Cognitive Sciences, lack of reasoning than by motivated reasoning. Cognition,
20(9), 689–700. https://doi.org/10.1016/j.tics.2016.07. 188, 39–50. https://doi.org/10.1016/j.cognition.2018.06.
001. 011.
Mercier, H., & Sperber, D. (2011). Why do humans reason? Pennycook, G., Ross, R. M., Koehler, D. J., & Fugelsang, J.
Arguments for an argumentative theory. Behavioral and A. (2016). Atheists and agnostics are more reflective than
Brain Sciences, 34(2), 57–74. https://doi.org/10.1017/ religious believers: Four empirical studies and a meta-
S0140525X10000968. analysis. Plos One, 11(4), e0153039. https://doi.org/10.
Metz, S. E., Weisberg, D. S., & Weisberg, M. (2018). Non- 1371/journal.pone.0153039.
Scientific criteria for belief sustain counter-scientific be- Perkins, D. (2019). Learning to reason: The influence of in-
liefs. Cognitive Science, 42(5), 1477–1503. https://doi. struction, prompts and scaffolding, metacognitive knowl-
org/10.1111/cogs.12584. edge, and general intelligence on informal reasoning about
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous everyday social and political issues. Judgment and De-
phenomenon in many guises. Review of General Psychol- cision Making, 14(6), 624–643. Retrieved from http://
ogy, 2(2), 175–220. https://doi.org/10.1037/1089-2680. journal.sjdm.org/19/190925a/jdm190925a.pdf.
2.2.175. Piaget, J. (1932). The moral judgment of the child. London,
Pennycook, G. (2018). The new reflectionism in cognitive UK: Routledge Kegan Paul.
psychology: Why reason matters. New York, NY: Rout- Price, E., Ottati, V., Wilson, C., & Kim, S. (2015). Open-
ledge. minded cognition. Personality and Social Psychology
Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & Bulletin, 41(11), 1488–1504. https://doi.org/10.1177/
Fugelsang, J. A. (2014a). Cognitive style and religiosity: 0146167215600528.
the role of conflict detection. Memory & Cognition, 42(1), Pronin, E., Lin, D. Y., & Ross, L. (2002). The bias blind
1–10. https://doi.org/10.3758/s13421-013-0340-7. spot: Perceptions of bias in self versus others. Personality
Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & and Social Psychology Bulletin, 28(3), 369–381. https://
Fugelsang, J. A. (2014b). The role of analytic thinking doi.org/10.1177/0146167202286008.
in moral judgements and values. Thinking & Reasoning, Redlawsk, D. (2002). Hot cognition or cool consideration?
20(2), 188–214. https://doi.org/10.1080/13546783.2013. Testing the effects of motivated reasoning on political
865000. decision making. Journal of Politics, 64, 1021–1044.
Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & https://doi.org/10.1111/1468-2508.00161.
Fugelsang, J. A. (2015). On the reception and detection of Redlawsk, D. P., Civettini, A. J. W., & Emmerson, K. M.
pseudo-profound bullshit. Judgment and Decision Mak- (2010). The affective tipping point: Do motivated reason-
ing, 10(6), 549–563. ers ever “get it”? Political Psychology, 31(4), 563–593.
Pennycook, G., Cheyne, J. A., Koehler, D. J., & Fugelsang, J. https://doi.org/10.1111/j.1467-9221.2010.00772.x.
A. (2016). Is the cognitive reflection test a measure of both Ross, L., & Ward, A. (1996). Naive realism in everyday life:
reflection and intuition? Behavior Research Methods, 48, Implications for social conflict and misunderstanding. In
341–348. https://doi.org/10.3758/s13428-015-0576-1. E. S. Reed, E. Turiel, & T. Brown (Eds.), The Jean Piaget
Pennycook, G., Cheyne, J. A., Seli, P., Koehler, D. J., symposium series. Values and knowledge (pp. 103–135).
& Fugelsang, J. A. (2012). Analytic cognitive style Hillsdale, NJ, US: Lawrence Erlbaum Associates, Inc.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press

 Judgment and Decision Making, Vol. 15, No. 4, July 2020 Actively open-minded thinking about evidence 498

Royzman, E. B., Landy, J. F., & Goodwin, G. P. (2014). Are Stanovich, K. E., West, R. F., & Toplak, M. E. (2013).
good reasoners more incest-friendly[202F?]? Trait cogni- Myside bias, rational thinking, and intelligence. Cur-
tive reflection predicts selective moralization in a sample rent Directions in Psychological Science, 22(4), 259–264.
of American adults. Judgment and Decision Making, 9(3), https://doi.org/10.1177/0963721413480174.
176–190. Strickland, A. A., Taber, C. S., & Lodge, M. (2011). Moti-
Rutjens, B. T., Sutton, R. M., & van der Lee, R. (2018). vated Reasoning and Public Opinion. Journal of Health
Not all skepticism is equal: Exploring the ideological an- Politics, Policy and Law, 36(6), 89–122. https://doi.org/
tecedents of science acceptance and rejection. Personality 10.1215/03616878-
and Social Psychology Bulletin, 44(3), 384–405. https:// Svedholm-Häkkinen, A. M., & Lindeman, M. (2018). Ac-
doi.org/10.1177/0146167217741314. tively open-minded thinking: development of a shortened
Sá, W. C., West, R. F., & Stanovich, K. E. (1999). The scale and disentangling attitudes towards knowledge and
domain specificity and generality of belief bias: Search- people. Thinking and Reasoning, 24, 21–40. https://doi.
ing for a generalizable critical thinking skill. Journal of org/10.1080/13546783.2017.1378723.
Educational Psychology, 91(3), 497–510. https://doi.org/ Svedholm, A. M., & Lindeman, M. (2013). The separate
10.1037/0022-0663.91.3.497. roles of the reflective mind and involuntary inhibitory
Sarathchandra, D., Navin, M. C., Largent, M. A., & Mc- control in gatekeeping paranormal beliefs and the under-
Cright, A. M. (2018). A survey instrument for measur- lying intuitive confusions. British Journal of Psychology,
ing vaccine acceptance. Preventive Medicine, 109, 1–7. 104(3), 303–319. https://doi.org/10.1111/j.2044-8295.
https://doi.org/10.1016/J.YPMED.2018.01.006. 2012.02118.x.
Shenhav, A., Rand, D. G., & Greene, J. D. (2012). Divine in- Swami, V., Voracek, M., Stieger, S., Tran, U. S., & Furnham,
tuition: Cognitive style influences belief in God. Journal A. (2014). Analytic thinking reduces belief in conspiracy
of Experimental Psychology. General, 141(3), 423–428. theories. Cognition, 133(3), 572–585. https://doi.org/10.
https://doi.org/10.1037/a0025391. 1016/j.cognition.2014.08.006.
Stanovich, K. E. (2005). The robot’s rebellion: Finding Thompson, V. A., Prowse Turner, J. A., & Pennycook, G.
meaning in the age of Darwin. Chicago, IL: Chicago (2011). Intuition, reason, and metacognition. Cognitive
University Press. Psychology, 63(3), 107–140. https://doi.org/10.1016/j.
Stanovich, K. E., & Toplak, M. E. (2019). The need for intel- cogpsych.2011.06.001.
lectual diversity in psychological science: Our own studies Thomson, K. S., & Oppenheimer, D. M. (2016). Investi-
of actively open-minded thinking as a case study. Cog- gating an alternate form of the cognitive reflection test.
nition, 187, 156–166. https://doi.org/10.1016/j.cognition. Judgment and Decision Making, 11(1), 99–113.
2019.03.006. Tobacyk, J. J. (2004). A revised paranormal belief scale.
Stanovich, K. E., & West, R. F. (1997). Reasoning indepen- The International Journal of Transpersonal Studies, 23(3),
dently of prior belief and individual differences in actively 94–98.
open-minded thinking. Journal of Educational Psychol- Toplak, M. E., West, R. F., & Stanovich, K. E. (2011). The
ogy, 89(2), 342–357. https://doi.org/10.1037/0022-0663. Cognitive Reflection Test as a predictor of performance on
89.2.342. heuristics-and-biases tasks. Memory & Cognition, 39(7),
Stanovich, K. E., & West, R. F. (1998). Individual differ- 1275–1289. https://doi.org/10.3758/s13421-011-0104-1.
ences in rational thought. Journal of Experimental Psy- White, K. R. G., Kinney, D., Danek, R. H., Smith, B., & Har-
chology: General, 127(2), 161–188. https://doi.org/10. ben, C. (2019). The Resistance to Change-Beliefs Scale:
1037/0096-3445.127.2.161. Validation of a new measure of conservative ideology.
Stanovich, K. E., & West, R. F. (2000). Individual dif- Personality and Social Psychology Bulletin. https://doi.
ferences in reasoning: implications for the rational- org/10.1177/0146167219841624.
ity debate? Behavioral and Brain Sciences, 23(5),
645–665; discussion 665-726. https://doi.org/10.1017/
S0140525X00003435.

https://doi.org/10.1017/S1930297500007439 Published online by Cambridge University Press