ORIGINAL RESEARCH

published: 30 January 2017

doi: 10.3389/fpsyg.2017.00048

Cognitive, Parent and Teacher Rating

Measures of Executive Functioning:
Shared and Unique Influences on
School Achievement
Marielle C. Dekker 1*, Tim B. Ziermans 1, 2 , Andrea M. Spruijt 1, 2 and Hanna Swaab 1, 2
1
Department of Clinical Child and Adolescent Studies, Faculty of Social and Behavioural Sciences, Leiden University, Leiden,
Netherlands, 2 Leiden Institute for Brain and Cognition, Leiden University, Leiden, Netherlands

Very little is known about the relative influence of cognitive performance-based executive
functioning (EF) measures and behavioral EF ratings in explaining differences in children’s
school achievement. This study examined the shared and unique influence of these
different EF measures on math and spelling outcome for a sample of 84 first and second
graders. Parents and teachers completed the Behavior Rating Inventory of Executive
Function (BRIEF), and children were tested with computer-based performance tests from
the Amsterdam Neuropsychological Tasks (ANT). Mixed-model hierarchical regression
analyses, including intelligence level and age, showed that cognitive performance and
Edited by: teacher’s ratings of working memory and shifting concurrently explained differences in
Mariette Huizinga,
spelling. However, teacher’s behavioral EF ratings did not explain any additional variance
VU University Amsterdam,
Netherlands in math outcome above cognitive EF performance. Parent’s behavioral EF ratings did not
Reviewed by: add any unique information for either outcome measure. This study provides support
Peter K. Isquith, for the ecological validity of performance- and teacher rating-based EF measures, and
Dartmouth College, USA
Michelle Ellefson,
shows that both measures could have a complementary role in identifying EF processes
University of Cambridge, UK underlying spelling achievement problems. The early identification of strengths and
*Correspondence: weaknesses of a child’s working memory and shifting capabilities, might help teachers
Marielle C. Dekker
to broaden their range of remedial intervention options to optimize school achievement.
[email protected]
Keywords: working memory, inhibition, shift, math, spelling
Specialty section:
This article was submitted to
Developmental Psychology, INTRODUCTION
a section of the journal
Frontiers in Psychology Executive functions (EFs) are generally defined as effortful cognitive abilities that help plan,
Received: 31 October 2016 guide and control goal-directed mental processes and behavior. Executive control is assumed
Accepted: 09 January 2017 to be involved in both math and spelling performance. Math calls for executive control to
Published: 30 January 2017 select and manipulate relevant numbers, to disregard irrelevant information, to choose the right
Citation: computational methods, to temporarily store and manipulate numbers and other information, and
Dekker MC, Ziermans TB, Spruijt AM to be able to switch between various procedures or operations (e.g., Raghubar et al., 2010; Friso-
and Swaab H (2017) Cognitive, Parent
van den Bos et al., 2013; Yeniad et al., 2013; Cragg and Gilmore, 2014). Written spelling requires
and Teacher Rating Measures of
Executive Functioning: Shared and
understanding in the language forms (i.e., morphology), sound structures, word meanings, and
Unique Influences on School origins. Written spelling is also assumed to require executive control in order to efficiently integrate
Achievement. Front. Psychol. 8:48. phonological, orthographical, and morphological information, and motor planning (Berninger
doi: 10.3389/fpsyg.2017.00048 et al., 2006; Garcia et al., 2010; Preßler et al., 2013).

Frontiers in Psychology | www.frontiersin.org 1 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

The observation that EF abilities mature at different rates over performance situations at school where factors like fear and
time and have their peaks at different ages, suggests that EF motivation also play an important role. Cognitive EF measures
incorporates separable abilities (e.g., Klenberg, 2001; Davidson tend to neglect the effects of motivation, goals, and beliefs on
et al., 2006; Simonds et al., 2007; Best et al., 2009; Best and EF, and their use in predicting quality of cognitive learning is
Miller, 2010). In many studies of school-aged children, there is an complicated by task impurity problems (Salthouse et al., 2003).
agreement that there are at least three fundamental EF abilities EF functioning is thought to be visible in everyday life whenever
that are interrelated, but distinguishable: working memory, planning, problem solving, inhibition or troubleshooting is
inhibitory control, and cognitive shifting or cognitive flexibility challenged. One might ask whether daily executive functioning
(e.g., Miyake et al., 2000; Jacob and Parkinson, 2015). at school or at home is also related to math and spelling
Working memory (WM) refers to the ability to temporarily performance. This would indicate the pervasive influence of EF
store, manipulate and control incoming information at the same on school performance on several levels of control.
time. WM improves gradually during childhood and adolescence Behavioral ratings of EF were developed to assess the
in a linear fashion (Best et al., 2009; Best and Miller, 2010). application of EF skills in typical performance situations at
Inhibitory control allows for the suppression of actions and home or at school and are assumed to be more ecologically
resistance to interference from irrelevant stimuli entering the valid. However, studies relating behavioral measures of EF to
WM and is considered to be a precondition for other EFs. school achievement are limited. A significant association between
During the preschool years, inhibition skills improve rapidly and behavioral WM problems and poorer math outcome has been
around age four children show basic inhibitory control. These reported by some (Clark et al., 2010; Gerst et al., 2015), but not
skills gradually and linearly improve between ages five to eight by others (Ten Eycke and Dewey, 2016). Behavioral inhibitory
and further refinements in accuracy and speed occur in middle problems have been found to show either a significant association
childhood and in adolescence (Best et al., 2009; Best and Miller, (Clark et al., 2010; Gerst et al., 2015) or no association with
2010; Clark et al., 2010). Shifting or cognitive flexibility refers math (Ten Eycke and Dewey, 2016). Behavioral problems with
to the ability to flexibly switch between strategies, rules, tasks or shifting have also been related to poorer math outcome (Gerst
mental states. Both WM and inhibition skills are needed to shift et al., 2015; Ten Eycke and Dewey, 2016). To our knowledge,
effectively and efficiently (Garon et al., 2008; Best and Miller, only one study reported on the association between spelling
2010). Shifting ability develops from preschool years through and behavioral EF (teacher report) and showed that behavioral
adolescence (Best et al., 2009; Best and Miller, 2010). aspects of memory, shifting, and inhibitory control were related
Most research on the influence of EF on school achievement to children’s spelling outcome in kindergarten and first grade
focuses on performance-based measures of EF (e.g., Allan et al., (Kent et al., 2014). Nevertheless, behavioral ratings are challenged
2014). Cognitive performance-based EF tasks tend to measure by rater bias (e.g., the halo effect, central tendency bias, leniency
the efficiency of information processing mechanisms of the brain. bias) and situational specificity of behavior, resulting in low cross-
WM capacities in children have been clearly linked to math skills informant agreement (Achenbach et al., 1987). Furthermore, the
(e.g., DeStefano and LeFevre, 2004; Raghubar et al., 2010; Friso- high correlations between the different subscales also point to
van den Bos et al., 2013; Gerst et al., 2015). In two meta-analyses, scale-impurity problems, questioning whether general behavioral
inhibitory control has also been positively linked to various math impairment is being measured rather than different aspects of
skills in preschoolers and kindergartners (Allan et al., 2014) executive dysfunctioning (McAuley et al., 2010).
and in primary school-aged children (Friso-van den Bos et al., Both cognitive performance-based EF measures and
2013), and also in recent studies a significant association between behavioral EF rating measures clearly have their pros and
inhibition and math performance has been found (e.g., Gerst cons. Results from a recent review study on the association
et al., 2015; Ten Eycke and Dewey, 2016). In two meta-analyses, between these EF measures in 13 studies using the Behavior
shifting was associated with math skills in primary school-aged Rating Inventory of Executive Function (BRIEF; Gioia et al.,
children (Friso-van den Bos et al., 2013; Yeniad et al., 2013). A 2000), showed that only 19% of the reported correlations were
recent study by Gerst et al. (2015) also reported a significant and significant with a median correlation of 0.18 (Toplak et al.,
positive relation between math and shifting. 2013). It is evident that measures assessing cognitive and
A varying amount of research has been performed on the behavioral EF across informants tap into different aspects of EF.
relation between cognitive measures of EF and spelling outcome, Meta-analytical evidence on inhibitory control in preschoolers
with most studies on WM, and only a few on inhibition or and kindergartners (Allan et al., 2014), showed that the mean
shifting. Studies on WM in relation to spelling skills show a association between math achievement and inhibition was
positive association (e.g., Jongejan et al., 2007; Malstädt et al., stronger for performance tasks (r = 0.35) compared to other-
2012; Cardoso et al., 2013; Fischbach et al., 2013; Preßler et al., reports (r = 0.22). However, it is not yet clear how these different
2013; Becker et al., 2014; Re et al., 2014; Bexkens et al., 2015). EF measures concurrently relate to real world external criteria
Both inhibition (Altemeier et al., 2008) and shifting (Altemeier like school achievement. Understanding to what extent different
et al., 2008) have also been positively linked to spelling in first to EF measures share variance and add unique variance in relation
fourth graders. Although cognitive EF performance is associated to school achievement could verify their validity and could
to cognitive performance in math and spelling, it remains unclear provide us with a more balanced view of relevant EF aspects.
whether cognitive measures of EF are the best option to explain Thus far, only the studies of Gerst et al. (2015) and Miranda
the more complicated, more demanding, and less structured et al. (2015) provide some insight into the relative impact of

Frontiers in Psychology | www.frontiersin.org 2 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

these different types of EF measures on school outcome, although rating-based measure would add unique variance. Based on Gerst
math outcome was only studied by Gerst et al. (2015) and et al. (2015) findings on reading comprehension, we tentatively
neither of these two studies looked at spelling. Gerst et al. (2015) assumed that behavioral EF ratings would have the biggest impact
examined both cognitive EF measures and teacher behavioral on our language related spelling outcome, except for WM for
EF rating measures of WM, inhibition and shifting and found which the cognitive measure was also expected to add unique
moderate correlations for all measures with math and reading variance. We further assumed that teacher’s ratings of EF would
comprehension outcome. Analyzing the shared and unique have a bigger association with school achievement than parent’s
influence of these cognitive and behavioral measures for each EF EF ratings (Miranda et al., 2015), as EF demands at home are
in a full model with relevant covariates showed that both types different then EF demands at school, with the latter being more
of WM measures were complementary in the prediction of math likely to be related to school readiness, attitude toward learning
and reading comprehension outcome. However, for inhibition and testing, and thus with school achievement.
and shifting, the behavioral EF rating did not add any unique
variance to the prediction of math by the performance measure. METHODS
In contrast, for reading comprehension, the cognitive measures
for inhibition and shifting did not add any unique variance to Procedure
the teacher rating. Miranda et al. (2015) concluded that teacher’s The current study is part of an ongoing pretest-posttest
global EF rating was more strongly related to reading accuracy intervention study called “Curious Minds’ that focuses on
and speed then parent’s global EF rating. neurocognitive, social, and environmental factors affecting
A key issue when examining the impact of EF on school children’s” learning at school and at home. Children were
achievement is to what extent it is independent from intelligence recruited from two primary schools in the Dutch province of
(IQ). There is some evidence that IQ has associations with Zuid-Holland during November 2013 (school 1) and March 2014
WM (Mahone et al., 2002; Friedman et al., 2006; Alloway and (school 2). The Ethical Board of the department of Education and
Alloway, 2010), inhibition (Mahone et al., 2002) and shifting Child studies at Leiden University has given ethical approval for
(Ardila et al., 2000; van der Sluis et al., 2007), and that this this study (ECPW-2010016).
relationship is partially attributable to shared executive or non- Only children in grade 1 or 2, all aged 6–8 years, were included
executive processing demands (e.g., processing speed) underlying in this study. All parents of students from grade 1 or 2 (N = 172)
both EF and IQ assessment (van der Sluis et al., 2007), as well as received written information about the study from their child’s
to shared method variance reflected in the ability to take tests school and were invited to attend an informational meeting.
in the case of performance based EF tasks. Some studies did Written informed consent was obtained from all 105 parents
indeed show that EF shared a lot of variance with IQ in predicting who participated (response = 61.0%). Chi-square tests with a
school achievement (e.g., Bull and Scerif, 2001; Espy, 2004). continuity correction showed no significant differences between
However, other studies, have shown that both performance-based participants and non-participants in gender, grade, or school (all
and rating-based EF measures were uniquely related to school p > 0.05), neither did a t-test for age (p > 0.05).
achievement after taking into account the possible confounding All parents and teachers were asked to complete a
effects of intelligence (e.g., George and Greenfield, 2005; Alloway questionnaire on their child’s or student’s behavioral EF.
and Alloway, 2010; Preßler et al., 2013; Yeniad et al., 2013; Gerst Cognitive EF data was collected during school visits. Each
et al., 2015; Dekker et al., 2016). These latter findings suggest child completed several computer-based performance-based
that traditional intelligence tests might not assess abilities that EF tasks. Each assessment period lasted about an hour and a
are considered important from a neurocognitive perspective, half and took place in a quiet room to minimize distraction. All
and that IQ cannot be considered a proxy of EF or vice versa. assessments were done by the researchers or by Master’s students
However, the mixed findings point to the need to study the who completed an extensive training in test administration,
possible confounding effect of intelligence level. including video-feedback sessions. Pretest data was collected in
The aim of this study was to examine the shared and the period between November 2013 and February 2014 (school
unique influence of three different types of EF measures, i.e., 1), and May and June 2014 (school 2). Intelligence level was
performance-based, teacher’s rating-based, and parent’s rating- assessed during the post-test data collection phase. As IQ is
based, on math and spelling outcome in first and second considered to be quite stable over time, we expected that the time
graders, while taking level of intelligence into account. Based between this study’s pre- and post-test of about half a year, would
on the presented evidence we expected cognitive measures be of negligible influence (Canivez and Watkins, 1998). Dutch
of WM, inhibition and shifting to be related to math and standardized paper-and-pencil achievement tests scores used to
spelling. Because there are only a couple of studies, with monitor math and spelling progress were retrieved from each
contradicting results, concerning behavioral EF measures as school’s records at pretest. We obtained full achievement test
markers for math and spelling differences, our expectations were score information, full cognitive EF data and teacher EF ratings
tentative. Nevertheless, we assumed that behavioral executive for 104 out of the 105 participating children, for 103 children we
dysfunctioning had a negative association with math and spelling were able to estimate intelligence level, and we received 86 EF
outcome. Based on the findings of Gerst et al. (2015), we expected ratings from parents. Complete data for this study was available
cognitive measures of EF to have the biggest impact on math for 84 children (80.0% of all participating children; 48.8% of
outcome, except for WM where we predicted the behavioral all eligible children) from 7 different classes. Children with

Frontiers in Psychology | www.frontiersin.org 3 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

complete data did not significantly differ from children without opposite side of where a red ball is presented). This part consists
complete data (N = 21) on age, grade, school or gender (all p > of 80 trials; 40 trials requiring a compatible response and 40 trials
0.05). requiring an incompatible response. The overall amount of errors
in part 3 was used to measure level of visuospatial shifting.
Measures
Cognitive EF Behavioral EF
Cognitive EF was measured with three neuropsychological Behavioral EF was measured with BRIEF (Gioia et al., 2000;
tasks from the Amsterdam Neuropsychological Tasks (ANT, Huizinga and Smidts, 2009, 2010). Both the teacher’s form
version 2.0; De Sonneville, 1999, 2011). The ANT has been (BRIEF-teacher) and the parent’s form (BRIEF-parent) were
used extensively to examine EF and related cognitive processes used. The BRIEF teacher’s form assesses everyday behavioral EF
in various clinical and non-clinical populations and has high problems in the classroom and the BRIEF parent’s version does
sensitivity for neuropsychological problems as well as good the same for the home situation. Fifteen different classroom
reliability and appropriate validity (De Sonneville, 2005, 2014; teachers filled out 5–9 BRIEF-teacher questionnaires (mean
Rowbotham et al., 2009). All computer tasks were preceded = 5.6; mode = 4; SD = 1.6). The BRIEF has satisfactory
by instructions from the test leader and practice trials. All test internal consistency, test-retest reliability, moderate inter-
stimuli were presented on a computer screen and the child had to rater agreement and appropriate evidence of predictive and
respond by pressing a mouse key. discriminant validity and is used for children from 5 to 18 years
old. The BRIEF contains 86 items that make up eight scales that
Working memory form a Behavior Regulation Index. In this study we used the
Visuospatial working memory was measured with the ANT raw scale score of the Working Memory, the Inhibit, and the
Spatial Temporal Span (STS–part 2)—backward span. In this Shift scale. A higher BRIEF scale score indicates a higher level
task, nine squares are presented on the computer screen in of executive dysfunction.
a three-by-three matrix. During each trial, an incremental
sequence of these squares (two up to a maximum of nine) is Problems with working memory
pointed out by a hand animation. Each sequence of appointed The Working Memory scale (WM) of the BRIEF assesses the
squares is presented in two successive trials. The participant capability to hold information when completing a task, when
is instructed to repeat this sequence by clicking the same encoding information, or when generating goals/plans in a
squares in reverse order. In each trial the sequence is preceded sequential manner (e.g., forgets what he/she was doing, trouble
by an auditory cue (a beep). The task aborts automatically remembering things, losing track of what they are doing).
whenever two successive trials of the same sequence number
are incorrect. The number of correct identified targets in correct Problems with inhibitory control
order backwards was used as a measure of visuospatial working The Inhibit scale of the BRIEF assesses the amount of trouble a
memory. child has controlling impulses and to stop engaging in a behavior
(e.g., gets out of control more than friends, has difficulty staying
Inhibition seated in the classroom, often interrupts others in class, requires
Inhibition of a prepotent ongoing motor response was assessed more adult supervision).
with the ANT Go-NoGo (GNG–biased) task. In the GNG task
the mouse button has to be clicked whenever a yellow square Problems with shifting
with a hole at the bottom is displayed (the Go signal; 75% of The Shift scale of the BRIEF assesses the problems a child has
the trials). Whenever a full yellow square is displayed (the NoGo with moving freely from one activity or situation to another,
signal; 25% of the trials) the child has to withhold the prepotent alternating attention or changing strategies (e.g., difficulty to
motor response and do nothing. The number of false alarms on flexibly solve problems, to make transitions, tolerate change, or
the 18 NoGo trials was used as a measure of level of inhibition. shift attention).
A higher amount of false alarms (e.g., the participant clicks when
the target signal is not presented) indicates that a child is less able Intelligence Level
to stop an ongoing response. Level of intelligence (IQ) was estimated using the Vocabulary
(V) and Block Design (BD) subtest of the Dutch Wechsler
Shifting Intelligence Scale for Children 6–17 years old (WISC-III-NL) at
Shifting was assessed with the ANT Response Organization post-test, about half year later (Kort et al., 2005). The short form
Objects (ROO–part 3)—mixed compatible and incompatible. estimates of full scale IQ for the WISC-III (FSIQ) were obtained
During the third part of the ROO task, the color of the ball according to the algorithm: 2.9 × (sum of normed scores) +
alternates randomly between green and red and the child has 42; an algorithm based on Tellegen and Briggs’s linear scaling
to shift between response sets. Whenever the green ball appears technique (Tellegen and Briggs, 1967; Campbell, 1998). The
a compatible dominant response is required (click the mouse WISC-III V-BD estimate has been found valid for the estimation
button that corresponds to the side where the green ball is of full scale IQ, given a sufficient corrected FSIQ validity (r =
presented) and when the red ball appears an incompatible 0.82) and split-half reliability (r = 0.91) (Campbell, 1998). The
subdominant response is required (click the mouse button on the 2.8 year stability of the WISC-III Vocabulary subtest has been

Frontiers in Psychology | www.frontiersin.org 4 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

found to be 0.75, and that of Block Design subtest 0.78 (Canivez three methods (cognitive, teacher rating, and parent rating), and
and Watkins, 1998). including IQ. A random intercept for class (n = 7) was included
to control for the slight non independence of our data due to
School Achievement students being nested in classes (multi-level data). The intra class
To assess math and spelling ability we used the Dutch standard correlation (ICC = Variance (intercept)/(Variance(intercept) +
CITO Mathematics Test (CMT; Janssen et al., 2010) and CITO Variance(error)) for the null model (intercept-only model) of
Spelling Test (CST; de Wijs et al., 2010). The CMT and the math was 0.03 (3% of the variance was attributed to class level)
CST are both composite national curriculum paper-and-pencil and for spelling the ICC was 0.08. The difference in −2Log
achievement tests that are standardized and norm-referenced. Likelihood, which follows a χ 2 distribution with the difference
They have good psychometric properties and are commonly used in degrees of freedom between the two nested models as its
in Dutch schools to monitor the progress of students in primary degrees of freedom, between two adjacent nested models was
education (de Wijs et al., 2010; Jansen et al., 2013). There are two calculated and also the Schwarz’s Bayesian Information Criterion
different tests for each grade, one regularly administered halfway (BIC) difference. A BIC difference between two nested models
through the year (January) and one around June. We collected can be considered a weak (0–2), a positive (2–6) or a strong
the CMT and CST scores through the schools at the time of the (>6) indication for a better model (Raftery, 1995). A model was
pretest. Therefore, in this study we used the January 2014 CITO considered an improved model whenever the −2LL difference
tests scores from school 1, and the June 2014 CITO tests score was significant (p < 0.05) and the BIC difference was bigger than
from school 2. To allow for comparison between the students’ 0. In each hierarchical model, IQ was entered first (model 1). For
math and spelling scores we used the age equivalent math score math outcome, the next model included the cognitive EF measure
(AES) and subtracted the number of months of education the (Model 2). If this model was a significant improvement over
student had received up to that point (10 months per year, the IQ only model, a model adding the corresponding teachers’
starting from grade 1). A positive score of 5 means that a student EF rating was estimated (Model 3). The matching parent’s EF
is about 5 months ahead in mathematical or spelling skills relative rating was entered after the teacher’s rating (Model 4). For
to the amount of education received up to that point in time (the spelling outcome, Model 2 included the teacher’s EF rating. If
general population AES mean is 0 months). this model was a significant improvement over the IQ only
model, a model adding the corresponding cognitive EF measure
Mathematical abilities was estimated (Model 3). The matching parent’s EF rating was
The Dutch standard CITO Mathematics test (CMT) was used entered after the cognitive EF measure (Model 4). Whenever an
to assess various mathematical abilities (Janssen et al., 2010). In EF measure would not significantly improve a previous model,
the current study’s grades the following math skills are covered: we would replace this measure with the next EF counterpart
(a) number and number relations; (b) addition and subtraction; measure (adding b or c to the model name). As only a small
(c) multiplication and division; and (d) measuring (e.g., weights, pool of not substantially correlated independent variables (see
length, surface, time). Table 2) were included in this study, we also ran a mixed-model
stepwise backwards regression analyses. As similar results were
Spelling abilities found when using this method of model selection, we only report
The Dutch standard CITO Spelling test (CST) was used to assess the hierarchical approach estimates in this paper, including
implicit spelling abilities (de Wijs et al., 2010). Spelling ability for fixed effect (intercept, regression weights) and the random effect
the current study’s age group is tested by having children write estimates (variance around the intercept and random error).
50 words (January Grade 1) or sentences (June grade 1) dictated Effect sizes were interpreted as: I. a small ‘practically’ significant
by their teacher. Starting from grade 2 there are two parts: (1) effect (r or β ≥ 0.2 and <0.5); II. a moderate effect (r or β ≥ 0.5
25 dictated sentences; and (2). 25 questions where children have and <0.8) or III. a strong effect (r or β ≥ 0.8) (Ferguson, 2009).
to pick out the sentence with the wrongly spelled word (in bold
case) out of four different sentences. All CST scores are rescaled
RESULTS
to make the CST comparable across children.
Sample Description
Statistical Analysis Sample characteristics are shown in Table 1. Age (range 6–8
Data was analyzed using simple correlations and with linear years) and gender (51.1% male) distributions were as expected.
mixed-effects modeling using IBM SPSS version 23. All variables Children in this study were on average around 2 months ahead
that were significantly skewed (SE > 3.0) were first log in math and spelling compared to a norm sample of Dutch
transformed (BRIEF Inhibit and Shift scale for both parent and peers, and had a somewhat higher estimated mean IQ score
teacher rating) or square root transformed (GNG number of of 106. Comparing the educational level of the 164 parents
false alarms, ROO number of errors part 3, BRIEF WM scale in our sample to the educational level of the general Dutch
for both parent and teacher rating). A hierarchical mixed-model population of 25- to 45-year-olds (N = 4,267,000), showed
regression analysis, based on our hypotheses, with maximum that the parents in our study were less likely to have a low
likelihood estimation was used to test each hypothesized model educational level (11.6 vs. 33.6%; z = −5.96, p < 0.001),
explaining math or spelling achievement outcome. Analysis were were more likely to have a medium educational level (48.8 vs.
performed for each type of EF (WM, inhibition, shifting) using all 28.3%; z = 5.83; p < 0.001), and equally likely to have a high

Frontiers in Psychology | www.frontiersin.org 5 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

TABLE 1 | Demographic characteristics and descriptive statistics of educational level (39.6 vs. 38.1%; z = 0.40; p = 0.689) (CBS,
independent and dependent variables. 2013). Around 12% of the children were referred to mental health
% (N = 84) Mean (SD) Scale range
care in the past year (95% Confidence Interval = 5.0–18.8%)
sample
for the assessment and/or treatment of various developmental,
emotional and behavioral problems (e.g., problems with attention
DEMOGRAPHIC VARIABLES and hyperactivity, anxiety, conduct related problems, pervasive
Age† 87.54 (7.16) months 75–102 months developmental problems). This percentage is significantly higher
First grade 56.0 than the 5.9% referral rate found in a large (N = 1710) Dutch
School 1 67.9 general population study of 6–18-year-olds (z = −2.23, p =
Males 48.9 0.026) (Tick et al., 2008). Teachers in our sample scored their
Educational level students significantly more often in the clinical range of WM
parents‡ problems (T-score ≥ 65 = 20.2%) compared to 7% of the BRIEF-
High 39.6 teacher Dutch norm sample of 5- to 8-year-olds (N = 55) (Z-
Medium 48.8 score = −2.138, p = 0.032). No significant difference with the
Low 11.6 Dutch norm sample on the percentage of reported students in
Mental Health Care 11.9 the clinical range was found for inhibition and shifting. Parents
referral past year in our sample reported a similar percentage of children in the
DEPENDENT VARIABLES¶ clinical range on all three BRIEF-parent scales compared to the
Math 2.67 (7.87) months −22 to 26 months Dutch BRIEF-parent norm sample of 5- to 8-year-olds (N = 311;
Spelling 2.27 (6.45) months −18 to 17 months all p > 0.05).
INDEPENDENT VARIABLES
Full scale IQ estimate§ 106.11 (12.17) 79.70–131.90 Correlations between EF, IQ, and School
% < 85/% > 115 4.8/25.0
Achievement
Cognitive EF measures: ANT (scale)
Correlations between all measures are reported in Table 2. Both
STS working 31.45 (14.59) 4–75
standardized measures of math and spelling were significantly
memory (0–88)
correlated with all three types of WM measures (|r| range = 0.28–
GNG inhibit (0–18) 3.58 (2.49) 0–11
0.43), which were significantly interrelated amongst themselves
ROO-3 shift (0–80) 9.17 (9.48) 0–35
as well (|r| range = 0.25–0.31). Math and spelling were also
Teacher behavioral rating scales: BRIEF-teacher (scale 10–30)
significantly associated with the cognitive shifting measure, as
Raw score working 15.63 (5.12) 10–29
was spelling with the teacher shifting problems rating. All
memory
effects were within the small range. None of the inhibition
% T-score ≥ 65 20.2
measures were related to school achievement. Parent-teacher
Raw score inhibit 13.50 (4.46) 10–30
cross-informant agreement of similar EFs were all significant and
% T-score ≥ 65 10.7
within the small range, while the cross-informant correlations
Raw score shift 13.71 (3.70) 10–28
between different types of EF were higher and in the moderate
% T-score ≥ 65 17.9
range. Intelligence level was significantly associated with math
Parent behavioral rating scales: BRIEF-PARENT (scale 10–30) achievement (r = 0.41) and with the teacher’s rating of WM
Raw score working 15.74 (4.44) 10–29 problems (r = −0.31), but not with spelling achievement or any
memory
of the other EF measures. Furthermore, no significant correlation
% T-score ≥ 65 4.8
between age with any of the EF variables was found in this sample
Raw score inhibit 15.73 (4.18) 10–30
of 6–8 year olds.
% T-score ≥ 65 6.0
Raw score shift 12.18 (3.21) 10–29
% T-score ≥ 65 6.0
Math Achievement: Shared and Unique
Influence of EF Measures
‡
†
At time of Standardized CITO Math and Spelling test. % based on N = 164 parents In the best mixed models explaining math achievement
using the Standard Classification of Education (SOI) 2006, edition 2014/15: “Low (see Table 3), standardized math achievement was uniquely
educational level (1),” including Primary and Lower secondary education (level 1 and 2 of
the SOI); “Medium educational level (2),” including Upper secondary and Post-secondary
associated with intelligence level (b∗ ranging from 0.34 to 0.38),
non-tertiary education (level 3 and 4 of the SOI); “High educational level (3),” including the cognitive measure of WM (b∗ (number correct) = 0.35),
Short cycle tertiary education and Bachelor’s, Master’s and Doctoral level (level 5–8 of and the cognitive measure of shifting (b∗ (number of errors) =
the SOI; Dutch Central Bureau for Statistics [CBS], 2006, 2011). ¶ Difference between
−0.22), all with an effect size within the small range (see Table 3).
achievement level (expressed as equivalent to number of months of education) and
number of moths of education (10 months per grade). § The short form (Vocabulary and
None of the inhibition measures had a direct impact on math
Block Design) estimates of full scale IQ for the Wechsler Intelligence Scale For Children for achievement. None of the teacher’s or the parent’s EF ratings
children aged 6–8 years old (WISC-III-NL; Kort et al., 2005) were obtained according added any unique variance to their cognitive EF counterpart in
to the algorithm: 2.9 x (sum of normed scores) + 42 (Campbell, 1998). STS, Spatial
relation to math achievement. As age was uncorrelated with any
Temporal Span (raw score number of identified targets in correct order backwards); GNG,
raw score number of false alarms–biased; ROO-3, raw score number of errors compatible of the outcome or the EF measures (see Table 2), including age in
and incompatible part 3; BRIEF, Behavior Rating Inventory of Executive Function. the analysis did not make a difference to the final results. Similar

Frontiers in Psychology | www.frontiersin.org 6 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

TABLE 2 | Correlations between IQ estimate, executive function measures and standardized test scores for math and spelling (N = 84).

Cognitive EF Measures Behavioral EF Scales IQ School Achievement Age

BRIEF-parent BRIEF-teacher

Cognitive EF Measures WM Inhibit I Shift WM Inhibit Shift WM Inhibit Shift Math Spelling

WM (STS) 0.09 −0.25* −0.25* −0.06 −0.15 −0.25* 0.04 −0.10 0.20 0.43** 0.37** 0.18
Inhibit I (GNG) 0.01 0.03 0.10 −0.07 −0.05 0.01 −0.09 0.04 0.15 0.03 −0.08
Shift (ROO-3) 0.19 0.03 0.10 0.19 0.10 −0.01 −0.19 −0.27* −0.24* 0.03
Behavioral EF Scales: BRIEF-parent
WM 0.60** 0.42** 0.31** 0.30** 0.27** −0.12 −0.28** −0.25* 0.18
Inhibit 0.60** 0.14 0.41** 0.31** 0.05 −0.08 −0.14 0.19
Shift 0.06 0.19 0.31** −0.01 −0.11 −0.08 0.19
Behavioral EF Scales: BRIEF-teacher
WM 0.57** 0.64** −0.31** −0.23* −0.37* −0.09
Inhibit 0.57** −0.06 0.04 −0.09 −0.04
Shift −0.07 −0.13 −0.24* −0.06

IQ 0.41** 0.17 −0.18

Math 0.34** −0.07
Spelling −0.06

*p <0.05; **p <0.001. WM, working memory; BRIEF, Behavior Rating Inventory of Executive Function; STS, Spatial Temporal Span (number of identified targets in correct order
backwards); GNG, Go-NoGo; ROO-3, Response Organisation Objects-part 3; Bold, monotrait–heteromethod correlations; Italic, heterotrait–monomethod correlations; regular,
heterotrait–heteromethod correlations.

results for EF on math were found when IQ was excluded from EF performance measures to explain differences in school
the analysis, showing somewhat higher standardized regression achievement. Furthermore, little research on the relation between
weights for WM (b∗ = 0.43) and shifting (b∗ = −0.29), as shared EF and spelling has been published, especially in typically
variance with IQ was not corrected for. developing children using multiple modes of EF assessments.
Analyses included IQ, a confounding factor for both school
Spelling Achievement: Shared and Unique achievement and EF.
The main findings of this study were that the cognitive WM
Influence of EF Measures measure was correlated with its parent- and teacher-reported
The best mixed models for spelling outcome (see Table 4),
behavioral WM counterpart, and that all WM measures were
showed that both teacher rated WM problems (b∗ = −0.34)
significantly associated with school achievement. Furthermore,
and the cognitive WM measure (b∗ (number correct) = 0.29)
both the cognitive shifting and the teacher-reported behavioral
uniquely explained differences in spelling achievement, while IQ
shifting measure were also related to school achievement.
did not. A similar result was found for shifting, with both teacher
None of the inhibition measures were significantly correlated
rated problems with shifting (b∗ = −0.24) and the cognitive
with school outcome. Moderate correspondence was observed
shifting measure (b∗ (number of errors) = −0.27) accounting
between parent’s and teacher’s ratings of children’s behavioral EF.
for spelling differences. All effects sizes were within the small
Cognitive performance and teacher’s ratings of WM and shifting
range. None of the inhibition measures were related to spelling
concurrently explained differences in spelling achievement.
achievement, neither were any of the parent EF ratings. As age
However, teacher’s behavioral EF ratings did not explain any
was uncorrelated with any of the outcome or the EF measures (see
additional variance in math outcome above IQ and cognitive
Table 2), including age in the analysis did not make a difference
EF performance. Parent’s behavioral EF ratings did not add any
to the final results. Excluding IQ from of the model resulted in
unique information to either outcome measure.
similar findings for EF with regard to spelling achievement.
In comparing similar cognitive and behavioral aspects of
EF, a significant and modest monotrait-multimethod correlation
CONCLUSION AND DISCUSSION was only found between cognitive and behavioral ratings
of WM. Thus, visual spatial working memory performance
The aim of the present study was to develop a better was somewhat linked to real-life WM problems that were
understanding of the interrelations between cognitive EF observed by others, like forgetting what one was doing and
measures and behavioral EF ratings from both parents and having trouble remembering things at school or at home.
teachers and to investigate their shared and unique influence Furthermore, modest correlations between parent and teacher
on math and spelling achievement in first and second ratings across all comparable EFs were found. These modest
graders. A novel aspect of this study is the inclusion of EF relations were consistent with findings by Toplak et al. (2013)
ratings from multiple informants concurrently with cognitive and cross-informant findings in the related field of child

Frontiers in Psychology | www.frontiersin.org 7 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

TABLE 3 | Mixed model hierarchical regression analyses results of best model explaining MATH outcome (N = 84) for each type of EF using multiple
methods, IQ, and with random intercept for class (n = 7).

Independent −2LL (df) Nested 1-2LL p (1 nested model) MATH: final model estimates
variables within BIC 1 BIC
nested models† Fixed effects Random effects

b (SE) b* p Var intercept p Var error p

0. Null model 583.51 (3) 2.66 (1.00) 0.037 2.03 (4.5) 0.617 59.15 (9.58) <0.001
ICC = 0.03 596.80
WM
Intercept −26.88 (6.29) <0.001 1.76 (3.05) 0.564 41.59 (6.74) <0.001
1. IQ 567.04 (4) 16.47 <0.001 0.22 (0.06) 0.34 <0.001
584.77 12.03
2. STS 554.38 (5) 12.66 <0.001 0.19 (0.05) 0.35 <0.001
576.53 8.24
3a. BRIEF-t 554.15 (6) 0.23 0.632
580.74
3b. BRIEF-p 550.67 (6) 3.74 0.053
577.26 −0.73
INHIBITION
Intercept −25.99 (6.76) <0.001 3.23 (4.14) 0.435 47.62 (7.71) <0.001
1. IQ 567.04 (4) 16.47 <0.001 0.27 (0.06) 0.42 <0.001
584.77 12.03
2a. GNG 564.66 (5) 2.38 0.123
586.81 −2.04
2b. BRIEF-t 566.60 (5) 0.44 0.507
588.75 −3.98
2c. BRIEF-p 566.34 (5) 0.70 0.403
588.50 −3.73
SHIFTING
Intercept −19.66 (7.07) <0.001 4.90 (4.85) 0.312 43.85 (7.09) <0.001
1. IQ 567.04 (4) 16.47 <0.001 0.24 (0.06) 0.38 <0.001
584.77 12.03
2. ROO-3 561.88 (5) 5.16 0.023 −4.37 (1.86) −0.22 0.021
584.03 0.74
3a. BRIEF-t 560.75 (6) 1.13 0.288
587.33 −3.30
3b. BRIEF-p 560.91 (6) 0.97 0.325
587.49 −3.46

†
Whenever difference −2LL between fuller model minus adjacent nested more parsimonious model (lower number) = significant and BIC difference > 0, fixed and random estimates
of best model are reported. ICC, intra class correlation; ∆-2RLL, −2Log Likelihood difference between two adjacent nested models (∆ df = difference in degrees of freedom between
two adjacent nested models) following χ2 distribution; ∆ BIC, difference in Schwarz’s Bayesian Criterion between two adjacent nested models; p (∆ nested model), significance level
improvement of adjacent more parsimonious model; b, regression weight; SE, Standard Error; b*, standardized regression weight; Var(intercept), variance attributed to class; Var(error),
random error; WM, Working Memory; STS, Spatial Temporal Span; GNG, Go-NoGo; ROO-3, Response Organization Objects -part 3; BRIEF-p, Behavior Rating Inventory of Executive
Functioning–parent rating; BRIEF-t, BRIEF-teacher rating.

psychopathology (Achenbach et al., 1987). Teachers perceived that each type of EF measure taps different aspects of EF across
on average similar amounts of EF problem behavior compared different situations and under variable conditions. Furthermore,
to parents, but they only modestly agreed on which children the similar or even higher multitrait-monomethod correlations
had relatively more or less EF problems. This was also true for point to method variance caused by rater biases, e.g., halo and
reporting the presence of a clinical level of EF problems (T-score leniency bias, and test impurity problems.
≥ 65). Teachers in our sample were, compared to a norm sample
of peers, more likely to report a clinical level of EF problems Math Achievement
than parents did; this was especially true for WM. The observed Based on the presented evidence we expected cognitive measures
absent or modest monotrait-multimethod correlations suggest of WM, inhibition and shifting to be correlated to math

Frontiers in Psychology | www.frontiersin.org 8 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

TABLE 4 | Mixed model hierarchical regression analyses results of best model explaining SPELLING outcome (n = 84) for each type of EF using multiple
methods, IQ, and with random intercept for class (n = 7).

Independent −2LL (df) Nested 1-2LL p (1 nested model) SPELLING: final model estimates
variables within BIC 1 BIC
nested models† Fixed effects Random effects

b (SE) b* p Var intercept p Var error p

0. Null Model 548.31 (3) 2.21 (0.96) 0.054 3.24 (3.37) 0.335 37.75 (6.07) <0.001
ICC = 0.08 561.60
WM
Intercept 7.58 (8.00) 0.346 4.29 (3.57) 0.229 27.81 (4.48) <0.001
IQ‡ 544.23 (4) 4.08 0.043 0.03 (0.05) 0.06 0.527
561.96 −0.36
2. BRIEF-t 532.53 (5) 11.7 <0.001 −3.28 (1.09) −0.32 0.003
554.69 7.29
3. STS 525.00 (6) 7.53 0.006 0.12 (0.04) 0.28 0.006
551.58 3.11
4. BRIEF-p 522.89 (7) 2.11 0.146
553.91 −2.33
INHIBITION
Intercept
1. IQ 544.23 (4) 4.08 0.043
561.96 −0.36
2a. BRIEF-t 543.56 (5) 0.67 0.413
565.72 −3.76
2b. GNG 544.23 (5) 0.00 1.00
566.39 −4.43
2c. BRIEF-p 542.06 (5) 2.17 0.141
564.21 −2.25
SHIFTING
Intercept 11.64 (9.67) 0.232 4.20 (3.67) 0.252 31.37 (5.05) <0.001
1. IQ‡ 544.23 (4) 4.08 0.043 0.08 (0.05) 0.16 0.119
561.96 −0.36
2. BRIEF-t 540.30 (5) 3.93 0.047 −12.31 (6.37) −0.22 0.040
562.46 0.50
3. ROO-3 534.49 (6) 5.81 0.016 −3.89 (1.57) −0.24 0.016
561.08 0.88
4. BRIEF-p 534.45 (7) 0.04 0.841
564.46 −3.38

†
Whenever difference −2LL between fuller model minus adjacent nested more parsimonious model (= lower number) = significant and BIC difference > 0, fixed and random estimates
‡
of best model are reported. IQ is left in model to control for confounding even though BIC < 0. ICC = intra class correlation; ∆−2RLL = −2Log Likelihood difference between
two adjacent nested models (∆ df = difference in degrees of freedom between two adjacent nested models) following χ 2 distribution; ∆ BIC, difference in Schwarz’s Bayesian
Criterion between two adjacent nested models; p (∆ nested model), significance level improvement of adjacent more parsimonious model; b, regression weight; SE, Standard Error;
b*, standardized regression weight; Var(intercept), variance attributed to class; Var(error), random error; WM, Working Memory; STS, Spatial Temporal Span; GNG, Go-NoGo; ROO-3,
Response Organization Objects-part 3; BRIEF-p, Behavior Rating Inventory of Executive Functioning–parent rating; BRIEF-t, BRIEF–teacher rating.

achievement (e.g., Yeniad et al., 2013; Friso-van den Bos et al., relation to math, and that inhibition and shifting showed the
2013; Gerst et al., 2015; Ten Eycke and Dewey, 2016). Our study weakest relation. Furthermore, our findings also differed from
confirmed these findings, except for inhibition. Our finding that previous findings linking inhibition to emerging math skills in
inhibition did not have a direct relation with math was in contrast preschoolers and kindergartners (e.g., Espy et al., 2004; Blair and
to findings from a meta-analysis of 4–12-year-old children (Friso- Razza, 2007; Allan et al., 2014). Perhaps, only more extreme
van den Bos et al., 2013), and from recent studies in 9- to levels of inhibitory problems affect math outcome negatively,
11-year-olds (Gerst et al., 2015), and in 5–18 year-olds (Ten or inhibition is more likely to play a role in children with
Eycke and Dewey, 2016), although the meta-analysis of Friso- mathematical disorders or from economically disadvantaged
van den Bos et al. (2013) also showed that WM had the strongest families, which were included in the meta-analyses of Allan et al.

Frontiers in Psychology | www.frontiersin.org 9 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

(2014) and Friso-van den Bos et al. (2013). In fact, the meta- by Gerst et al. (2015) on another language related outcome,
analysis of Friso-van den Bos et al. (2013) showed that children i.e., reading comprehension. In our study we found that both
with math, psychological or physical problems have stronger teacher behavioral ratings and cognitive measures of WM and
associations between EF and math outcome. The children in our shifting were related to spelling outcome, partially confirming
study were not at risk for mathematical problems nor inhibition our tentative hypotheses. Thus, real life application of WM
problems, and predominantly came from families with medium and shifting skills at school helps to explain differences in
to high socio-economic backgrounds. This study also showed spelling outcome concurrently with their cognitive counterparts.
that the influence of EF on math is in addition to the effect of Spelling in this study was assessed through a dictation test, which
IQ, which is in line with previous research (e.g., George and might ask for different EF skills compared to a general math
Greenfield, 2005; Alloway and Alloway, 2010; Preßler et al., 2013; achievement test, although in first grade the math questions
Yeniad et al., 2013; Gerst et al., 2015; Dekker et al., 2016), and were also read out loud by the teacher. Perhaps attentional
underscores the suggestion that IQ cannot be considered a proxy processes play a bigger role during dictation tests. Indeed, parent
of EF or vice versa. and teacher ratings of inattention in children with emotional
Based on the study of Gerst et al. (2015), we expected that only and behavioral problems have previously been associated with
for WM a behavioral measure, most likely the teacher’s rating, behavioral EF ratings on the BRIEF (McAuley et al., 2010),
would add unique variance to the cognitive WM measure and which might partially explain the contribution of behavioral EF
IQ in explaining math performance. Unlike Gerst et al. (2015), ratings concurrently with cognitive EF measures in explaining
we did not observe a similar impact for the teacher WM rating, differences in spelling outcome.
nor for the parent rating of WM, although the latter measure was In sum, although the ecological validity of cognitive
borderline significant. Nevertheless, comparable to Gerst et al. performance-based tasks have been questioned, this study
(2015), our results showed that none of the behavioral measures confirmed that cognitive EF measures actually explained most
of inhibition or shifting added any unique variance explaining unique variance in math outcome compared to behavioral EF
math outcome besides IQ. Thus, for math achievement we were measures. This study also provides support for the ecological
able to confirm most of Gerst et al. (2015) findings in a younger validity of performance- and teacher rating-based EF tasks by
age group, while also including parent EF ratings. showing that both measures have a complementary role in
identifying spelling achievement problems. Furthermore, both
Spelling Achievement WM and shifting abilities were related to school achievement in
Based on research about the relation between EF and spelling, general rather than to a specific domain.
we expected the cognitive measure of WM to be related to Several study limitations need to be acknowledged. First of
spelling outcome (e.g., Fischbach et al., 2013; Preßler et al., all, children from only two Dutch schools in the same provincial
2013; Becker et al., 2014). We could confirm that WM was region were included in this study. One school from a rural
related to spelling performance. Our results also extend the area and a second school from a town that is part of the
previous finding by Altemeier et al. (2008) that in typically metropolis of the cities of Rotterdam and The Hague. Although
developing first to fourth graders shifting ability is related to the distributions of our independent and outcome measures seem
spelling, although we could not confirm their finding of a to represent levels of typically developing children, with the
significant relation between inhibition and spelling. Inhibition exception of teacher reported level of clinical WM problems, it
and emerging writing skills have also been linked in preschoolers is clear that the children in our study are not representative as far
(Blair and Razza, 2007; McClelland et al., 2007; Brock et al., as the educational level of their parents is concerned. Children
2009). Altemeier et al. (2008) used a verbal word-color naming from parents with a low educational level are underrepresented,
task to assess inhibition and shifting, while in our study we and our results cannot be generalized to this group. Our low
used nonverbal tasks. Perhaps, measures of verbal inhibition risk sample might have resulted in weaker relations between
have a stronger association with spelling skills than non-verbal EF and school achievement than those found in other studies
measures. Research in math, for example, has shown that visual comprising at-risk samples (e.g., Waber et al., 2006; Gerst et al.,
spatial WM is more strongly related to learning something 2015). Stronger associations between EF and math outcome
new, while verbal working memory is more related to learned exist in children with relatively more math, psychological or
math skills, which are typically evaluated through standardized physical problems, as was shown in the meta-analysis of Friso-
achievement tests that are also used in this study (Van de Weijer- van den Bos et al. (2013). Secondly, the inclusion of more classes
Bergsma et al., 2015). Similar differences across different stages from more schools would have given more reliable estimates of
of spelling attainment might also be observed for inhibition. random variation around the intercept for class. Thirdly, this
Future research is needed to address the relative impact of verbal study used a cross-sectional design, so we could not study the
vs. visual spatial performance based EF measures in relation to differential predictive power of the various EF measures nor
various school outcomes and taking into account different stages the development of EF in relation to school achievement over
of the learning process (e.g., acquiring or mastering). time, which precludes any causal inferences. Finally, it might be
No previous publications have considered the joint impact of possible that the inclusion of teacher-based math and spelling
different EF measures on spelling. We based our expectations, grades could have resulted in a different pattern of the relative
i.e., teacher’s EF ratings having the biggest influence, and the contribution of each type of EF measure, as grades might share
cognitive measure of WM also adding variance, on the findings more variance with behavioral measures.

Frontiers in Psychology | www.frontiersin.org 10 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

Despite these limitations, the observation that WM and it might be informative to study independent aspects of
shifting were related to spelling and math outcome, regardless math (e.g., factual, procedural, conceptual; Raghubar et al.,
of the child’s IQ level, points in the direction of possible benefits 2010).
from stimulating EF skills in young children in addition to extra
domain specific instruction, to optimize school performance. ETHICS STATEMENT
There is some evidence that school-based and computerized
interventions aimed at improving EF skills have promising This study was carried out in accordance with the
cognitive outcomes in young children (Thorell et al., 2009; recommendations of the standards of the Ethical Committee
Diamond and Lee, 2011; Diamond, 2012; Wass, 2015), although of the Leiden Institute of Education and Child Studies with
questions remain concerning the actual causal mechanisms written informed consent from the parents of all subjects
involved in improving school achievement. For example: To (minors). All parents of subjects (minors), gave written informed
what extent do these interventions directly train academic consent in accordance with the Declaration of Helsinki. The
achievement? Or to what level do these interventions improve protocol was approved by the Ethics Committee of the Leiden
EF by reducing EF suppressors like anxiety, depressive feelings, Institute of Education and Child Studies at Leiden University
sleep deprivation or low physical activity level? (Jacob and (ECPW-2010/016).
Parkinson, 2015; Diamond and Ling, 2016). Other remaining
questions are the transfer of EF skills, the heterogeneity AUTHOR CONTRIBUTIONS
or homogeneity of the training regime, how long benefits
last, and which children benefit the most. There is some MD was involved in the conception and design of the work, data
indication that younger children and children from at risk collection, data analysis and interpretation, drafting the article,
groups (e.g., economically disadvantaged background, poor critical revision of the article and gave her final approval of the
EF) benefit more from EF training (Diamond, 2012; Wass, version to be published. TZ was involved in data interpretation,
2015). Nevertheless, identifying and monitoring each child’s EF critical revision of the article and gave his final approval of
strengths and weaknesses, especially in the WM and shifting the version to be published. AS was involved in the design of
domain might help teachers and other caregivers to broaden the work, data collection, data interpretation, critical revision
their range of remedial intervention options to optimize school of the article and gave her final approval of the version to
achievement. This study’s findings also show that both types of EF be published. HS was initiator of the Curious Minds study
measures, cognitive performance tasks and teacher’s behavioral and was involved as project leader in the conception and
rating scales, complement each other in explaining spelling design of this work, data interpretation, critical revision of
achievement and suggest that both could be used to identify likely the article and gave her final approval of the version to be
candidates for additional support. published.
Future research is needed to cross-validate our final models,
and to compare the impact of each type of EF measure across a FUNDING
wider age range of students, preferably longitudinally, to detect
developmental differences, and across more school achievement This research is funded by the Curious Minds Program, which
domains, using both verbal and non-verbal cognitive EF is supported by the Dutch Ministry of Education, Culture and
measures. Also, within certain domains, e.g., mathematics, Science and the National Platform Science & Technology.

REFERENCES Becker, D. R., Miao, A., Duncan, R., and McClelland, M. M. (2014).
Behavioral self-regulation and executive function both predict visuomotor
Achenbach, T. M., McConaughy, S. H., and Howell, C. T. (1987). Child/adolescent skills and early academic achievement. Early Childh. Res. Q. 29, 411–424.
behavioral and emotional problems: implications of cross-informant doi: 10.1016/j.ecresq.2014.04.014
correlations for situational specificity. Psychol. Bull. 101, 213–232. Berninger, V. W., Abbott, R. D., Thomson, J., Wagner, R., Swanson, H. L.,
doi: 10.1037/00332909.101.2.213 Wijsman, E. M., et al. (2006). Modeling phonological core deficits within
Allan, N. P., Hume, L. E., Allan, D. M., Farrington, A. L., and Lonigan, C. J. (2014). a working memory architecture in children and adults with developmental
Relations between inhibitory control and the development of academic skills dyslexia. Sci. Stud. Read. 10, 165–198. doi: 10.1207/s1532799xssr1002_3
in preschool and kindergarten: a meta-analysis. Dev. Psychol. 50, 2368–2379. Best, J. R., and Miller, P. H. (2010). A developmental perspective on executive
doi: 10.1037/a003749310.1037/a0037493 function. Child Dev. 81, 1641–1660. doi: 10.1111/j.14678624.2010.01499.x
Alloway, T. P., and Alloway, R. G. (2010). Investigating the predictive roles of Best, J. R., Miller, P. H., and Jones, L. L. (2009). Executive functions after age 5:
working memory and IQ in academic attainment. J. Exp. Child Psychol. 106, changes and correlates. Dev. Rev. 29, 180–200. doi: 10.1016/j.dr.2009.05.002
20–29. doi: 10.1016/j.jecp.2009.11.003 Bexkens, A., van den Wilden‘berg, W. P. M., and Tijms, J. (2015). Rapid
Altemeier, L. E., Abbott, R. D., and Berninger, V. W. (2008). Executive functions automatized naming in children with dyslexia: is inhibitory control involved?
for reading and writing in typical literacy development and dyslexia. J. Clin. Dyslexia 21, 212–234. doi: 10.1002/dys.1487
Exp. Neuropsychol. 30, 588–606. doi: 10.1080/13803390701562818 Blair, C., and Razza, R. P. (2007). Relating effortful control, executive
Ardila, A., Pineda, D., and Rosselli, M. (2000). Correlation between intelligence test function, and false belief understanding to emerging math and
scores and executive function measures. Arch. Clin. Neuropsychol. 15, 31–36. literacy ability in Kindergarten. Child Develop. 78, 647–663.
doi: 10.1016/S0887-6177(98)00159-0 doi: 10.1111/j.1467-8624.2007.01019.x

Frontiers in Psychology | www.frontiersin.org 11 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

Brock, L. L., Rimm-Kaufman, S. E., Nathanson, L., and Grimm, K. J. (2009). Espy, K. A., McDiarmid, M. M., Cwik, M. F., Stalets, M. M., Hamby, A., and
The contributions of ‘hot’ and ‘cool’ executive function to children’s Senn, T. E. (2004). The contribution of executive functions to emergent
academic achievement, learning-related behaviors, and engagement in mathematic skills in preschool children. Dev. Neuropsychol. 26, 465–486.
kindergarten. Early Childh. Res. Q. 24, 337–349. doi: 10.1016/j.ecresq.2009. doi: 10.1207/s15326942dn2601_6
06.001 Ferguson, C. J. (2009). An effect size primer: a guide for clinicians and
Bull, R., and Scerif, G. (2001). Executive functioning as a predictor of children’s researchers. Prof. Psychol. Res. Pract. 40, 532–538. doi: 10.1037/a00
mathematics ability: inhibition, switching, and working memory. Develop. 15808
Neuropsychol. 19, 273–293. doi: 10.1207/S15326942DN1903_3 Fischbach, A., Könen, T., Rietz, C. S., and Hasselhorn, M. (2013). What
Campbell, J. M. (1998). Internal and external validity of seven Wechsler is not working in working memory of children with literacy disorders?
Intelligence Scale for Children—Third Edition short forms in Evidence from a three-year-longitudinal study. Read. Writ. 27, 267–286.
a sample of psychiatric inpatients. Psychol. Assess. 10, 431–434. doi: 10.1007/s11145-013-9444-5
doi: 10.1037/10403590.10.4.431 Friedman, N. P., Miyake, A., Corley, R. P., Young, S. E., DeFries, J. C., and Hewitt,
Canivez, G. L., and Watkins, M. W. (1998). Long-term stability of the Wechsler J. K. (2006). Not all executive functions are related to intelligence. Psychol. Sci.
Intelligence Scale for Children—Third Edition. Psychol. Assess. 10, 285–291. 17, 172–179. doi: 10.1111/j.1467-9280.2006.01681.x
doi: 10.1037/1040-3590.10.3.285 Friso-van den Bos, I., van der Ven, S. H. G., Kroesbergen, E. H., and van
Cardoso, A. M., Silva, M. M., and Pereira, M. M. (2013). Phonological awareness Luit, J. E. H. (2013). Working memory and mathematics in primary school
and the working memory of children with and without literacy difficulties. children: a meta-analysis. Educ. Res. Rev. 10, 29–44. doi: 10.1016/j.edurev.2013.
Codas 25, 110–114. doi: 10.1590/S2317-17822013000200004 05.003
CBS (2013). Beroepsbevolking; Behaalde Onderwijs Naar Persoonskenmerken Garcia, N. P., Abbott, R. D., and Berninger, V. W. (2010). Predicting poor, average,
2001-2012 [Working Population; Educational Attainment by Personal and superior spellers in grades 1 to 6 from phonological, orthographic, and
Characteristics 2001 - 20012]. Central Bureau of Statistics. Available online at: morphological, spelling, or reading composites. Written Lang. Lit. 13, 61–98.
http://statline.cbs.nl/Statweb/ doi: 10.1075/wll.13.1.03gar
Clark, C. A. C., Pritchard, V. E., and Woodward, L. J. (2010). Preschool executive Garon, N., Garon, S., and Bryson, I. (2008). Executive function in preschoolers:
functioning abilities predict early mathematics achievement. Dev. Psychol. 46, a review using an integrative framework. Psychol. Bull., 134, 31–60.
1176–1191. doi: 10.1037/a0019672 doi: 10.1037/0033-2909.134.1.31
Cragg, L., and Gilmore, C. (2014). Skills underlying mathematics: the role of George, J. L., and Greenfield, D. B. (2005). Examination of a structured problem-
executive function in the development of mathematics proficiency. Trends solving flexibility task for assessing approaches to learning in young children:
Neurosci. Educ. 3, 63–68. doi: 10.1016/j.tine.2013.12.001 relation to teacher ratings and children’s achievement. J. Appl. Dev. Psychol. 26,
Davidson, M. C., Amso, D., Anderson, L. C., and Diamond, A. (2006). 69–84. doi: 10.1016/j.appdev.2004.10.006
Development of cognitive control and executive functions from 4 to 13 years: Gerst, E. H., Cirino, P. T., Fletcher, J. M., and Yoshida, H. (2015).
evidence from manipulations of memory, inhibition, and task switching. Cognitive and behavioral rating measures of executive function as
Neuropsychologia 44, 2037–2078. doi: 10.1016/j.neuropsychologia.2006.02.006 predictors of academic outcomes in children. Child Neuropsychol.
Dekker, M. C., Ziermans, T. B., and Swaab, H. (2016). The impact of doi: 10.1080/09297049.2015.1120860. [Epub ahead of print].
behavioural executive functioning and intelligence on math abilities in Gioia, G. A., Isquith, P. K., Guy, S. C., and Kenworthy, L. (2000). Test
children with intellectual disabilities. J. Intellect. Disabil. Res. 60, 1086–1096. review behavior rating inventory of executive function. Child Neuropsychol. 6,
doi: 10.1111/jir.12276 235–238. doi: 10.1076/chin.6.3.235.3152
De Sonneville, L. (1999). “Amsterdam neuropsychological tasks: a computer- Huizinga, M., and Smidts, D. P. (2009). BRIEF Executieve Functies
aided assessment program,” in Cognitive Ergonomics, Clinical Assessment and Gedragsvragenlijst: Handleiding [Dutch Manual BRIEF Behavior Rating
Computer-Assisted Learning: Computers in Psychology, Vol. 6, eds B. P. L. M. Inventory of Executive Function]. Amsterdam: Hogrefe Uitgevers.
Den Brinker, P. J. Beek, A. N. Brand, S. J. Maarse, and L. J. M. Mulder (Lisse: Huizinga, M., and Smidts, D. P. (2010). Age-related changes in executive
Swets & Zeitlinger), 187–203. function: a normative study with the Dutch Version of the Behavior Rating
De Sonneville, L. (2011). Handleiding Amsterdamse Neuropsychologische Taken Inventory of Executive Function (BRIEF). Child Neuropsychol. 17, 51–66.
[Dutch Manual Amsterdam Neuropsychological Tasks]. Amsterdam: Boom Test doi: 10.1080/09297049.2010.509715
Uitgevers. Jacob, R., and Parkinson, J. (2015). The potential for school-based interventions
De Sonneville, L. (2014). Handboek Amsterdamse Neuropsychologische Taken that target executive function to improve academic achievement: a review. Rev.
[Dutch Handbook Amsterdam Neuropsychological Tasks]. Amsterdam: Boom Educ. Res. 85, 512–552. doi: 10.3102/0034654314561338
Test Uitgevers. Jansen, B. R. J., De Lange, E., and Van der Molen, M. J. (2013). Math practice
De Sonneville, L. M. J. (2005). Amsterdamse Neuropsychologische and its influence on math skills and executive functions in adolescents with
Taken: Wetenschappelijke en klinische toepassingen [Amsterdam mild to borderline intellectual disability. Res. Dev. Disabil. 34, 1815–1824.
Neuropscychological Tasks: Scientific and clinical applications]. Tijdschrift doi: 10.1016/j.ridd.2013.02.022
Voor Neuropsychologie 0, 27–41. Janssen, J., Verhelst, N., Engelen, R., and Scheltens, F. (2010). Wetenschappelijke
DeStefano, D., and LeFevre, J. A. (2004). The role of working memory in mental Verantwoording Van de Toetsen LOVS Rekenen-Wiskunde Voor Groep 3 tot en
arithmetic. Eur. J. Cogn. Psychol. 16, 353–386. doi: 10.1080/09541440244000328 Met Groep 8 [Scientific Justification of the Mathematics Test for Grade 1 until
de Wijs, A., Kamphuis, F., Kleintjes, F., and Tomesen, M. (2010). Grade 6]. Arnhem: Cito.
Wetenschappelijke Verantwoording Spelling Voor Groep 3 tot en Met 6 Jongejan, W., Verhoeven, L., and Siegel, L. S. (2007). Predictors of reading and
[Scientific Justification Spelling Grade 1 to 4]. Arnhem: CITO. spelling abilities in first- and second-language learners. J. Educ. Psychol. 99,
Diamond, A. (2012). Activities and Programs That Improve Children’s Executive 835–851. doi: 10.1037/0022-0663.99.4.835
Functions. Curr. Dir. Psychol. Sci. 21, 335–341. doi: 10.1177/0963721412453722 Kent, S., Wanzek, J., Petscher, Y., Al Otaiba, S., and Kim, Y.-S. (2014).
Diamond, A., and Lee, K. (2011). Interventions shown to aid executive Writing fluency and quality in kindergarten and first grade: the role of
function development in children 4 to 12 years old. Science 333, 959–964. attention, reading, transcription, and oral language. Read. Writ. 27, 1163–1188.
doi: 10.1126/science.1204529 doi: 10.1007/s11145-013-9480-1
Diamond, A., and Ling, D. S. (2016). Conclusions about interventions, programs, Klenberg, L. M.-N. P. (2001). Differential development of attention and executive
and approaches for improving executive functions that appear justified and functions in 3- to 12-year-old finnish children. Dev. Neuropsychol. 20, 407–428.
those that, despite much hype, do not. Dev. Cogn. Neurosci. 18, 34–48. doi: 10.1207/S15326942DN2001_6
doi: 10.1016/j.dcn.2015.11.005 Kort, W., Schittekatte, M., Dekker, P. H., Verhaeghe, P., Compaan, E. L., Bosmans,
Espy, K. A. (2004). Using developmental, cognitive, and neuroscience approaches M., et al. (2005). Manual Wechsler Intelligence Scale for Children - Dutch edition
to understand executive control in young children. Dev. Neuropsychol. 26, (WISC-III-NL), 3rd Edn. Amsterdam: Harcourt Test Publishers/Nederlands
379–384. doi: 10.1207/s15326942dn2601_1 Instituut voor Psychologen.

Frontiers in Psychology | www.frontiersin.org 12 January 2017 | Volume 8 | Article 48

Dekker et al. EF Measures and School Achievement

Mahone, E. M., Hagelthorn, K. M., Cutting, L. E., Schuerholz, L. J., Pelletier, S. Tellegen, A., and Briggs, P. F. (1967). Old wine in new skins: grouping
F., Rawlins, C., et al. (2002). Effects of IQ on executive function measures in Wechsler Subtests Into New Scales. J. Consult. Psychol. 31, 499–506.
children with ADHD. Child Neuropsychol. 8:52. doi: 10.1076/chin.8.1.52.8719 doi: 10.1037/h0024963
Malstädt, N., Hasselhorn, M., and Lehmann, M. (2012). Free recall behaviour in Ten Eycke, K. D., and Dewey, D. (2016). Parent-report and performance-based
children with and without spelling impairment: the impact of working memory measures of executive function assess different constructs. Child Neuropsychol.
subcapacities. Dyslexia 18, 187–198. doi: 10.1002/dys.1446 22, 889–906. doi: 10.1080/09297049.2015.1065961
McAuley, T., Chen, S., Goos, L., Schachar, R., and Crosbie, J. (2010). Is the behavior Thorell, L. B., Lindqvist, S., Bergman Nutley, S., Bohlin, G., and Klingberg,
rating inventory of executive function more strongly associated with measures T. (2009). Training and transfer effects of executive functions in preschool
of impairment or executive function? J. Int. Neuropsychol. Soc. 16, 495–505. children. Dev. Sci. 12, 106–113. doi: 10.1111/j.1467-7687.2008.00745.x
doi: 10.1017/S1355617710000093 Tick, N. T., van der Ende, J., and Verhulst, F. C. (2008). Ten-year increase in
McClelland, M. M., Cameron, C. E., Connor, C. M., Farris, C. L., Jewkes, A. service use in the Dutch population. Eur. Child Adolesc. Psychiatry 17, 373–380.
M., and Morrison, F. J. (2007). Links between behavioral regulation and doi: 10.1007/s00787-008-0679-7
preschoolers’ literacy, vocabulary, and math skills. Dev. Psychol. 43, 947–959. Toplak, M. E., West, R. F., and Stanovich, K. E. (2013). Do performance-based
doi: 10.1037/0012-1649.43.4.947 measures and ratings of executive function assess the same construct? J. Child
Miranda, A., Colomer, C., Mercader, J., Fernández, M. I., and Presentación, M. J. Psychol. Psychiatry 54, 131–143. doi: 10.1111/jcpp.12001
(2015). Performance-based tests versus behavioral ratings in the assessment of van der Sluis, S., de Jong, P. F., and van der Leij, A. (2007). Executive functioning in
executive functioning in preschoolers: associations with ADHD symptoms and children, and its relations with reasoning, reading, and arithmetic. Intelligence
reading achievement. Front. Psychol. 6:545. doi: 10.3389/fpsyg.2015.00545 35, 427–449. doi: 10.1016/j.intell.2006.09.001
Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., and Van de Weijer-Bergsma, E., Kroesbergen, E. H., and Van Luit, J. E. H. (2015).
Wager, T. D. (2000). The unity and diversity of executive functions and their Verbal and visual-spatial working memory and mathematical ability in
contributions to complex “Frontal Lobe” tasks: a latent variable analysis. Cogn. different domains throughout primary school. Mem. Cognit. 43, 367–378.
Psychol. 41, 49–100. doi: 10.1006/cogp.1999.0734 doi: 10.3758/s13421-014-0480-4
Preßler, A.-L., Könen, T., Hasselhorn, M., and Krajewski, K. (2013). Cognitive Waber, D. P., Gerber, E. B., Turcios, V. Y., Wagner, E. R., and Forbes,
preconditions of early reading and spelling: a latent-variable approach with P. W. (2006). Executive functions and performance on high-stakes
longitudinal data. Read. Writ. 27, 383–406. doi: 10.1007/s11145-013-9449-0 testing in children from urban schools. Dev. Neuropsychol. 29, 459–477.
Raftery, A. E. (1995). Bayesian model selection in social research. Sociol. Methodol. doi: 10.1207/s15326942dn2903_5
25, 111–163. doi: 10.2307/271063 Wass, S. V. (2015). Applying cognitive training to target executive
Raghubar, K. P., Barnes, M. A., and Hecht, S. A. (2010). Working functions during early development. Child Neuropsychol. 21, 150–166.
memory and mathematics: a review of developmental, individual doi: 10.1080/09297049.2014.882888
difference, and cognitive approaches. Learn. Individ. Differ. 20, 110–122. Yeniad, N., Malda, M., Mesman, J., van IJzendoorn, M. H., and Pieper, S. (2013).
doi: 10.1016/j.lindif.2009.10.005 Shifting ability predicts math and reading performance in children: a meta-
Re, A. M., Mirandola, C., Esposito, S. S., and Capodieci, A. (2014). Spelling errors analytical study. Learn. Individ. Dif. 23, 1–9. doi: 10.1016/j.lindif.2012.10.004
among children with ADHD symptoms: the role of working memory. Res. Dev.
Disabil. 35, 2199–2204. doi: 10.1016/j.ridd.2014.05.010 Conflict of Interest Statement: The authors declare that the research was
Rowbotham, I., Pit-ten Cate, I. M., Sonuga-Barke, E. J. S., and Huijbregts, S. conducted in the absence of any commercial or financial relationships that could
C. J. (2009). Cognitive control in adolescents with neurofibromatosis type 1. be construed as a potential conflict of interest.
Neuropsychology 23, 50–60. doi: 10.1037/a0013927
Salthouse, T. A., Atkinson, T. M., and Berish, D. E. (2003). Executive functioning Copyright © 2017 Dekker, Ziermans, Spruijt and Swaab. This is an open-access
as a potential mediator of age-related cognitive decline in normal adults. J. Exp. article distributed under the terms of the Creative Commons Attribution License (CC
Psychol. Gen. 132, 566–594. doi: 10.1037/0096-3445.132.4.566 BY). The use, distribution or reproduction in other forums is permitted, provided the
Simonds, J., Kieras, J. E., Rueda, M. R., and Rothbart, M. K. (2007). Effortful original author(s) or licensor are credited and that the original publication in this
control, executive attention, and emotional regulation in 7–10-year-old journal is cited, in accordance with accepted academic practice. No use, distribution
children. Cogn. Dev. 22, 474–488. doi: 10.1016/j.cogdev.2007.08.009 or reproduction is permitted which does not comply with these terms.

Frontiers in Psychology | www.frontiersin.org 13 January 2017 | Volume 8 | Article 48