Ecological Modelling 411 (2019) 108815

Contents lists available at ScienceDirect

Ecological Modelling
journal homepage: www.elsevier.com/locate/ecolmodel

Importance of spatial predictor variable selection in machine learning T

applications – Moving from data reproduction to spatial prediction
⁎
Hanna Meyera, , Christoph Reudenbachb, Stephan Wöllauerb, Thomas Naussb
a
Institute for Geoinformatics, Westfälische Wilhelms-Universität Münster, Heisenbergstr. 2, 48149 Münster, Germany
b
Faculty of Geography, Philipps-Universität Marburg, Deutschhausstr. 10, 35037 Marburg, Germany

A R T I C LE I N FO A B S T R A C T

Keywords: Machine learning algorithms find frequent application in spatial prediction of biotic and abiotic environmental
Cross-validation variables. However, the characteristics of spatial data, especially spatial autocorrelation, are widely ignored. We
Environmental monitoring hypothesize that this is problematic and results in models that can reproduce training data but are unable to
Machine learning make spatial predictions beyond the locations of the training samples. We assume that not only spatial validation
Overfitting
strategies but also spatial variable selection is essential for reliable spatial predictions.
Random Forests
Remote sensing
We introduce two case studies that use remote sensing to predict land cover and the leaf area index for the
“Marburg Open Forest”, an open research and education site of Marburg University, Germany. We use the
machine learning algorithm Random Forests to train models using non-spatial and spatial cross-validation
strategies to understand how spatial variable selection affects the predictions.
Our findings confirm that spatial cross-validation is essential in preventing overoptimistic model perfor-
mance. We further show that highly autocorrelated predictors (such as geolocation variables, e.g. latitude,
longitude) can lead to considerable overfitting and result in models that can reproduce the training data but fail
in making spatial predictions. The problem becomes apparent in the visual assessment of the spatial predictions
that show clear artefacts that can be traced back to a misinterpretation of the spatially autocorrelated predictors
by the algorithm. Spatial variable selection could automatically detect and remove such variables that lead to
overfitting, resulting in reliable spatial prediction patterns and improved statistical spatial model performance.
We conclude that in addition to spatial validation, a spatial variable selection must be considered in spatial
prediction models of ecological data to produce reliable results.

1. Introduction model is then used to make predictions in space, i.e. beyond the loca-
tions used for model training.
A key task in ecology is studying the spatial or spatio-temporal Most contemporary predictive modelling approaches use flexible
patterns of ecosystem variables, e.g. climate dynamics (Appelhans machine learning algorithms, which can approximate the nonlinear and
et al., 2015), variability of soil properties (Gasch et al., 2015) or dis- complex relationships found in nature. Recent software developments
tribution of vegetation types (Juel et al., 2015). Spatially continuous have simplified the application of machine learning algorithms (e.g. for
datasets of ecosystem variables are needed to analyze the spatial pat- R see Kuhn and Johnson, 2013). Noteworthy, however, is that machine
terns and dynamics. However, ecological variables are typically ac- learning is applied to ecological spatial modelling the same way as it is
quired through field work, which only provides data with a limited in other disciplines, while ignoring the unique characteristics of spatial
spatial extent, such as from climate stations, soil profiles or plot-based environmental data. Yet, spatial (and temporal) dependencies differ-
vegetation records. These data do not provide spatially continuous in- entiate spatial data from “ordinary” data and complicate the use of
formation about the variable of interest. Predictive modelling is a machine learning – due to the nature of the data, we cannot assume
method commonly used to derive spatially continuous datasets from samples are identically and independently distributed (i.i.d assumption)
limited field data (e.g. Lary et al., 2016). In predictive modelling, field (Xie et al., 2017). This is especially true when data are sampled in
data is used to train statistical models using spatially continuous pre- spatial clusters, which is a common design for providing ground truth
dictor variables derived from remote sensing imagery. The resulting data used in predictive modelling of ecological data.

⁎
Corresponding author.
E-mail address: [email protected] (H. Meyer).

https://doi.org/10.1016/j.ecolmodel.2019.108815
Received 26 March 2019; Received in revised form 29 August 2019; Accepted 29 August 2019
0304-3800/ © 2019 Elsevier B.V. All rights reserved.
H. Meyer, et al. Ecological Modelling 411 (2019) 108815

Previous studies in spatial applications of machine learning algo-

rithms have widely ignored the spatial dependencies in the data. One
problem of ignoring spatial dependencies in prediction methods be-
comes obvious in the error assessment of spatial predictive models.
Many authors have shown that the commonly used random cross-va-
lidation provides considerably overoptimistic error estimates due to the
problem of autocorrelation (Bahn and McGill, 2013; Micheletti et al.,
2014; Juel et al., 2015; Gasch et al., 2015; Gudmundsson and
Seneviratne, 2015; Roberts et al., 2017; Meyer et al., 2018). Hence
cross-validation strategies based on random data splitting fail to assess
a model's performance in terms of spatial mapping and only validate its
ability to reproduce the sampling data. Several methods for spatial
cross-validation have been proposed to account for spatial de-
pendencies in the data (Brenning, 2005; Le Rest et al., 2014; Fig. 1. Study area represented by the true color composite of the aerial image to
Pohjankukka et al., 2017; Roberts et al., 2017; Meyer et al., 2018; be classified. Polygons indicate the training areas of the different LULC classes
Valavi et al., 2018). While spatial cross-validation can provide objective used for model training. The yellow grid represents spatial blocks used for
and meaningful error estimates, the algorithms’ strong performance spatial cross-validation. Reference system: UTM 32N (WGS84). (For inter-
with random subsets and complete failures when predictions are made pretation of the references to color in this figure legend, the reader is referred to
beyond the spatial extent of the training samples still remains an issue. the web version of this article.)
Meyer et al. (2018) have shown for spatio-temporal data that spatial
(or spatio-temporal) dependencies can cause a misinterpretation of 2. Methods
certain predictor variables which makes flexible algorithms fail when
predicting beyond the location of the training data. Spatial de- The following sections describe the two case studies, the data, all
pendencies in predictor variables are most apparent in “geolocation” processing steps, modelling as well as validation. Data processing and
predictors that describe the spatial location of the training samples (e.g. modelling were performed in R Version 3.4 (R Core Team, 2018). The
coordinates, elevation, euclidean distances and all derivations of these scripts performing processing and analysis can be retrieved from
data). Hence, we assume that including predictor variables that de- https://github.com/HannaMeyer/EcoMod_SpML.
scribe the spatial location are problematic and prevent spatial models
from making meaningful contributions to ecological research. However,
2.1. Prediction task I: Land use/land cover classification
predictor variables that describe the spatial location rather than the
environmental properties are commonly included. Spatial coordinates
The first prediction task is to classify different types of forest, as well
are used especially often (Li et al., 2011; Langella et al., 2010; Shi et al.,
as adjacent LULC for the “Marburg Open Forest” http://nature40.org in
2015; Janatian et al., 2017; Walsh et al., 2017; Jing et al., 2016; Wang
Hessen, Germany. The basis for the classification is an aerial image that
et al., 2017; Georganos et al., 2019). Distances to certain points (e.g.
covers approx. 3000 × 2500 m (Fig. 1).
Hengl et al., 2018) or Euclidean distance to the corner coordinates of
the model domain (e.g. Behrens et al., 2018) have also been suggested
as predictors and included in models. 2.1.1. Reference data
This study uses autocorrelated spatial data to investigate the sen- A set of manually digitized polygons covering typical LULC classes
sitivity of machine learning applications to commonly applied geolo- are used as reference data which were selected by a combination of
cation predictors and shows pathways towards an automatic selection visual image inspection and knowledge firsthand from field work. In
of predictors that cannot be incorporated in spatial prediction tasks. We total, 10 different LULC classes were assigned (Table 1).
assume that spatial models cannot handle predictor variables that are
highly autocorrelated in space (e.g. geolocation) due to spatial de- 2.1.2. Predictor variables
pendencies in the training data. Algorithms can easily misinterpret such Spectral, terrain-related and geolocation variables were all prepared
variables, leading the model to make erroneous predictions outside of as potential predictor variables (Fig. 2). Spectral variables come from a
the locations of the training data. The problem becomes obvious in the 20 cm resolution aerial image (Hessische Verwaltung für
limited spatial performance of the model as well as in visually obvious Bodenmanagement und Geoinformation, 2018a) taken at the 30th of
artefacts in the spatial predictions. We therefore assume that spatial September 2015. For this study, the image was resampled to a spatial
variable selection is essential for automatically removing variables resolution of 1 m. The spectral predictors were the three channels of the
counterproductive to spatial mapping to provide scientifically valuable aerial image (red, green, blue). Further, the Visible Vegetation Index
results. (VVI, Planetary Habitability Laboratory, 2015), Triangular Greenness
We use two examples of classic prediction tasks in environmental
science to investigate our hypotheses. First, we perform a Land Use/ Table 1
Land Cover (LULC) classification, which is a common field for applying Summary of the different land use/land cover classes and the size of training
machine learning-based predictive modelling in the context of ecology data used for training of the classification model.
and remote sensing. The study area is located around the “Marburg Type Polygons Pixels
Open Forest”, an open research and education site owned by Marburg
University in Hessen, Germany. Second, we model the Leaf Area Index Beech 34 31,306
Douglas fir 20 13,241
(LAI) for the same region. Spectral, terrain-related as well as geoloca-
Field 40 59,663
tion variables are used as potential predictor variables in both ex- Grassland 85 27,134
amples. We study the effect of spatial and non-spatial cross-validation Larch 4 1568
on the estimated model performance with the frequently applied ma- Oak 23 17,804
chine learning algorithm Random Forest. A spatial variable selection is Road 38 18,461
Settlement 40 4722
suggested to analyze the importance of the potential predictor variables
Spruce 14 7521
for spatial mapping and their effect on the prediction outcomes. Water 66 3261

2
H. Meyer, et al. Ecological Modelling 411 (2019) 108815

Fig. 2. Example of spectral, terrain-related and geolocation

predictor variables: reflectance in the green band (green),
Visible Vegetation Index (VVI), standard deviation in a 9 × 9
pixel environment of the first Principal Component of all
spectral variables (PCA_9_sd), Digital Elevation Model (DEM),
latitude (Lat) and longitude (Lon). (For interpretation of the
references to color in this figure legend, the reader is referred
to the web version of this article.)

2.2. Prediction task II: Leaf Area Index modelling

The second prediction task aimed at modelling the LAI for the
forested area of the Marburg Open Forest, a classic example of a re-
gression task in environmental science.

2.2.1. Reference data

In this case study, the LAI reference was derived from lidar data
taken in the vegetation period 2010 (Hessische Verwaltung für
Bodenmanagement und Geoinformation, 2018b) that have 15 cm ver-
tical and 30 cm spatial accuracy. The LAI was calculated from the lidar
point cloud according to Getzin et al. (2017). Since no major man-
agement was present in the forest, the LAI from the lidar data was re-
garded as a reference for this study despite the time lag between lidar
Fig. 3. Study area represented by the true color composite of the Sentinel-2 derived reference and the Sentinel-2 based predictor variables. Espe-
scene, which serves as the baseline for the LAI predictions. Points represent the
cially since this study focus on the effect of validation and variable
location of the training samples. The color indicates the LAI values at these
selection strategies this time lag was neglected. The calculated LAI data
locations as derived from the lidar. The clear spatial clusters are the baseline for
the spatial cross-validation. Reference system: UTM 32N (WGS84). (For inter-
was then rasterized with 10 m spatial resolution to match the geometry
pretation of the references to color in this figure legend, the reader is referred to of the Sentinel-2 data that were used as predictors. To do this, the mean
the web version of this article.) of all LAI values located in the extent of a Sentinel-2 pixel was calcu-
lated. 11 spatially distinct clusters were then assigned in homogenous
areas of the forest. Every pixel in a 60 m radius around the center of
Index (TGI, Hunt et al., 2013), Normalized Green Red Difference Index
each cluster was used as training data, resulting in clear spatial clusters
(NGRDI), and Green Leaf Index (GLI, Hunt et al., 2013)) were derived
of training samples (Fig. 3). In total, 824 training pixels distributed
from the channels. A Principal Component Analysis (PCA) was per-
across the 11 clusters were used. The minimum, maximum and mean
formed on the three channels of the visible spectrum and the vegetation
LAI in this training data set were 0.9, 13.6 and 4.2, respectively. A LAI
indices; the first component of the PCA was included as an additional
below 1 means that the area is not fully covered by leaves. Values larger
potential predictor variable. In addition, the standard deviation of the
than 1 mean that more than one layer of leaves are present.
first principal component was calculated in 3x3 (PCA_3_sd), 5 × 5
(PCA_5_sd) and 9 × 9 (PCA_9_sd) pixel environments to account for the
spectral variability of LULC classes. A lidar-derived 1 m Digital Eleva-
2.2.2. Predictor variables
tion Model (DEM) was used as a terrain-related predictor. Elevation in
A Sentinel-2 scene as Level-1C product from 2017/05/10 was used
the study area ranges from 210 to 415 m. Slope and aspect were cal-
to derive spectral predictor variables. Sentinel-2 is the optical system
culated from the DEM in radians. The geolocation variables considered
from the earth observation mission from the EU Copernicus Programme
as potential predictors were latitude (Lat) and longitude (Lon). This
and has channels in the visible (bands 2–4), red edge (bands 5–7), near
results in a total set of 16 potential predictor variables for LULC pre-
infrared (band 8 and 8A) and short-wave infrared (bands 11 and 12)
diction.
part of the spectrum. The Sentinel-2 bands 1, 9 and 10 were not con-
sidered in this study because they do not include relevant information
for this prediction task. Hereafter, the used channels are referred to as
2.1.3. Compilation of the training data set
B01, B02, …, B12. Channels B02-B04 and B08 have a spatial resolution
The set of predictor variables was extracted for each training
of 10 m. The other channels have a resolution of 20 m and were re-
polygon. The model considered each pixel related to the polygons (e.g.
sampled to match the geometry of the 10 m channels. In addition to the
within, intersecting) as an individual training sample (Table 1). This
spectral channels, elevation, slope, aspect, as well as latitude and
resulted in a set of approx. 185,000 training samples. Each training
longitude (as described in the description of the previous prediction
sample contained the information about every potential predictor
task but resampled to a 10 m spatial resolution) were used as potential
variable as well as about the LULC class based on the information from
predictors. This results in a total set of 15 potential predictor variables
the polygons.
for LAI prediction.

3
H. Meyer, et al. Ecological Modelling 411 (2019) 108815

2.2.3. Compilation of the training data set cross-validation. Performance was estimated by random cross-va-
Values for each predictor variable was extracted from the locations lidation (see 2a in Fig. 4) and spatial cross-validation (see 2b in
of the training samples. Each training sample contained the extracted Fig. 4). The results of this model are used to show how “default”
information from all potential predictor variables as well as the in- variable selection affects the spatial model performance.
formation about the LAI based on the information from the lidar-de- 3. Model using selected variables only. Variable selection was based
rived reference points. on a forward feature selection with random cross-validation.
Performance was estimated by random cross-validation (see 3a in
2.3. Model training and prediction Fig. 4) and spatial cross-validation (see 3b in Fig. 4). The results of
this model are used to show how random variable selection affects
The Random Forest algorithm (Breiman, 2001) was chosen as the the spatial model performance.
machine learning algorithm for predictive modelling due to its promi- 4. Model using selected variables only. Variable selection was based
nence in ecological modelling. Random Forest bases on the concept of on a forward feature selection with spatial cross-validation.
regression and classification trees: a series of nested decision rules for Performance was estimated by random cross-validation (see 4a in
the predictors determine the response (also called reference, i.e. LULC Fig. 4) and spatial cross-validation (see 4b in Fig. 4). The results of
or LAI). Random forest repeatedly builds trees from random samples of this model are used to show how spatial variable selection affects
the training data. Each tree is a separate model of the ensemble. The the spatial model performance.
predictions of all trees are averaged to produce the final estimate. To
overcome correlation between trees, a number of predictors (mtry) are The following sections describe the different cross-validation and
randomly selected at each split. The best predictor from the random variable selection strategies in more detail.
subset is used at this split to partition the data.
In this study, the Random Forests implementation of the
randomForest package (Liaw and Wiener, 2002) in R was applied and 2.3.1. Cross-validation strategies
accessed via the caret package (Kuhn, 2016). Throughout the study, This study applied two cross-validation strategies: a standard
each Random Forest model consisted of 500 trees after no increase in random k-fold cross-validation and a spatial k-fold cross-validation.
performance could be observed using a higher number of trees. The Each strategy first splits the data into k folds and then repeatedly trains
number of randomly chosen predictor variables at each split of the tree the models (k times) using the data from all but one fold. The models
(“mtry”) was tuned between two and the number of predictor variables are evaluated based on how they perform with the left-out data (see
(16 for LULC predictions and 15 for LAI predictions). See Kuhn and Fig. 5). See (Kuhn and Johnson, 2013) for more detailed description on
Johnson (2013) for a more detailed description on the Random Forest cross-validation in general.
algorithm and mtry tuning. While the cross-validation procedure is the same for random and
To study the effect of spatial validation as well as spatial variable spatial cross-validation, the major difference is how the data points are
selection the following models were compared for both case studies: split into folds (see Fig. 5). For the standard random cross-validation,
each data point was randomly assigned to one of the k folds.
1. Model using all potential predictor variables. Performance was es- For the spatial cross-validation, we chose a spatial block approach
timated by random cross-validation (see 1a in Fig. 4). The results of as suggested in Roberts et al. (2017) and also Valavi et al. (2018) for the
this model are used to show the outcome of a “default” modelling LULC case study. Therefore, we divided the spatial domain into 20
approach. The performance was further estimated by spatial cross- equally sized spatial blocks (yellow grid in Fig. 1). For each training
validation (see 1b in Fig. 4). The results of this validation are used to sample, the spatial block affiliation was identified by the spatial loca-
show how spatial cross-validation affects the estimated error of a tion of the corresponding training polygon where the sample belongs
“default” model. to. If a training polygon lay within two spatial blocks, the sample was
2. Model using selected variables only. Variable selection was based only assigned to the one block in which the greater proportion of the
on the commonly used recursive feature elimination with spatial polygon lay. This precluded that training pixels from one (usually
homogeneous) polygon were present in two spatial blocks. Analogous

Fig. 4. Overview on the models compared in this study.

Spectral, terrain related as well as geolocation variables were
used as predictors (only examples shown here). The response
variable was derived from training polygons for the case study
of land cover classification or from lidar-derived Leaf Area
Index (LAI) values for the case study of LAI prediction. Models
were trained using either no variable selection or either re-
cursive feature elimination, default random forward feature
selection (FFS) or spatial FFS. Model performance was com-
pared with random cross-validation (CV) or spatial CV. The
entire modelling procedures were performed for the case
study of land cover classification as well as for prediction of
LAI.

4
H. Meyer, et al. Ecological Modelling 411 (2019) 108815

Fig. 5. Concept of random and spatial cross-validation (CV): A

total dataset (here: 9 different data points represented by
different shapes) is split into k folds (here: k = 3). Models are
then repeatedly trained by always leaving one of the folds out
and use it for model validation and not for model training.
Random CV means that the data are randomly split into folds.
Spatial CV means that the data are split into folds according to
spatial location (e.g. a spatial cluster or a spatial block, here
represented by unique color). Figure modified from Meyer
et al. (2018) and Kuhn and Johnson (2013). (For interpreta-
tion of the references to color in this figure legend, the reader
is referred to the web version of this article.)

to the random cross-validation, models were then repeatedly trained FFS, Fig. 4) to check that improvements were indeed due to the spatial
using data from all but one spatial block (= fold) and their performance selection and not the pure reduction of variables that lead to changes in
estimated using the left-out data, e.g. the spatial block left out of model performance. As FFS is very time consuming, mtry was not tuned but
training. Hence, models were assessed for their performance in making set to 2 for feature selection. Once the variables were selected, models
spatial predictions beyond the locations of the training data. For the were re-trained using the selected variables and either spatial or
case study of the LAI predictions, a leave-one-cluster-out cross-valida- random cross-validation with mtry being tuned between 2 and the
tion was applied. This method is similar to the concept described above number of selected predictor variables.
but instead of spatial blocks, in each iteration one of the 11 clusters In addition to FFS, we also used recursive feature elimination (RFE,
Fig. 3) was left out during model training. explained in Kuhn and Johnson, 2013) to compare state-of-the art
The number of spatial blocks or clusters, k, equaled the number of procedures (see e.g. Brungard et al., 2015; Meyer et al., 2017a,b; Ghosh
spatial blocks (20) or the number of spatial clusters (11), depending on and Joshi, 2014; Stevens et al., 2013, in the field of environmental
the case study. Random cross-validation was performed with the same mapping). However, we argue that the backward RFE selection fails to
number for k. address the issue of overfitting. RFE relies on variable importance
The validation measure for performance assessment for the LULC scores, which are only calculated using the training subset (Kuhn and
classification during cross-validation was the Kappa Index (Cohen, Johnson, 2013). If a variable leads to considerable overfitting, it is
1960) and the Accuracy. A Kappa of 0 (or lower) is associated with a highly significant in the models. Therefore, the RFE process will keep it
random classification result, while a Kappa of 1 indicates a perfect as important and not remove it, even if it results in a high spatial error.
classification. Since it accounts for chance agreement, it was used as the
prior validation measure for the classification task rather than the Ac- 3. Results
curacy. For the LAI predictions, the performance was assessed by the
Root-Mean-Square Error (RMSE) and the coefficient of determination 3.1. Statistical performance
(R2).
Models were compared by fold, which gives the average perfor- Using the “default” way of spatial prediction (using all potential
mance (e.g. Kappa) over all k folds from cross-validation. Models were variables) and the “default” random cross-validation, both Accuracy
also compared by their global performance, which results from a and Kappa index were higher than 0.99 for the classification task
comparison between every data point predicted during cross-validation, (Table 2). Random cross-validation indicated that the LAI could be
independently of the fold. predicted with a RMSE of 0.96 and a R2 of 0.87 (Table 3, Fig. 6a).
When these models were validated using a spatial cross-validation,
2.3.2. Spatial predictor variable selection the performance was considerably lower (Kappa value of 0.55 for the
Forward Feature Selection (FFS) as described in Meyer et al. (2018) LULC classification and RMSE of 1.25 for the LAI regression model,
and implemented in the CAST package (Meyer, 2018) for R was used for Table 3, Fig. 6b). A prominent source of high error estimates is that by
spatial variable selection. This FFS implementation works in conjunc- leaving entire clusters out for validation, a held back cluster that has
tion with user-defined cross-validation, hence it allows to select vari- higher LAI values than all other clusters could not adequately be
ables that lead to the highest spatial performance (if run in conjunction modelled since such high LAI values are unknown from the training
with spatial CV). First, the FFS trains models using every combination data (Fig. 6b). Note that low per-fold R2 values for the LAI regression
of two predictor variables. Based on the best performing variables (as models in Table 3 result from low variabilities within spatial folds
identified by cross-validation), FFS increases the number of variables to (Fig. 6b) so that the per-fold RMSE or especially the global R2/RMSE
test which (if any) variables further improve the (spatial) model's per- present the more reliable performance estimates here.
formance. Every variable that does not improve (or even decreases) the Using an RFE-based variable selection in conjunction with a spatial
model's performance is excluded. See Meyer et al. (2018) for a more cross-validation during the variable selection does not (LULC classifi-
detailed description on this FFS. cation Kappa = 0.55) or only marginally (LAI regression RMSE = 1.22)
This study used FFS with spatial cross-validation to test which improve the spatial performances. The same is true for an FFS-based
variables are significant to spatial mapping and which ones have no approach in conjunction with a random cross-validation during the
spatial meaning or are even counterproductive (spatial FFS, Fig. 4). For variable selection (Kappa = 0.14, RMSE = 1.23). In both cases, the
comparison, FFS was also run with random cross-validation (random random model performance stayed high (Kappa > 0.99 for both RFE

5
H. Meyer, et al. Ecological Modelling 411 (2019) 108815

Table 2 the latter.

Statistical performance of the models for LULC classification. Models were When FFS was paired with spatial cross-validation (spatial FFS) for
compared by fold, which gives the average performance over all k folds from the variable selection task, the spatial performance slightly improved
cross-validation (CV). Models were also compared by their global performance, (Kappa = 0.56, RMSE = 1.20) compared to all other models. It is no-
which is the Accuracy or Kappa for every data point predicted during cross-
teworthy that this type of variable selection reduces the model per-
validation. Bold numbers indicate a spatial validation that must be considered
formance indicators in a random cross-validation (Kappa = 0.87,
as the valid performance for the prediction task. For an overview on the model-
RMSE = 1.12) so that the differences in the error estimates between the
ID see also Fig. 4.
validation strategies became smaller. This validation was based on the
ID Variables CV By fold Global average performance for each fold that was left out during cross-vali-
dation. Similar patterns emerged when all independent predictions
Accuracy Kappa Accuracy Kappa
were simultaneously compared (global validation). Noticeable is that
1a All Random > 0.99 > 0.99 > 0.99 > 0.99 the R2 of LAI predictions increased from 0.58 (all variables, spatial
1b All Spatial 0.71 0.55 0.68 0.61 cross-validation) to 0.63 (spatial variable selection, spatial cross-vali-
2a Selected by RFE Random > 0.99 > 0.99 > 0.99 > 0.99
dation).
“spatial”
2b Selected by RFE Spatial 0.71 0.55 0.69 0.61
“spatial” 3.2. Variable importance and selected variables
3a Selected by FFS Random > 0.99 > 0.99 > 0.99 > 0.99
random
When all variables were presented to the algorithm to predict LULC
3b Selected by FFS Spatial 0.43 0.14 0.41 0.30
random and LAI, the most important variables were latitude, longitude and
4a Selected by FFS spatial Random 0.89 0.87 0.78 0.82 elevation (Fig. 7). The spectral predictor variables were considerably
4b Selected by FFS spatial Spatial 0.71 0.56 0.70 0.62 less important for these tasks.
The RFE based upon this variable importance ranking did not
eliminate any variables for the LULC classification, hence the model is
Table 3
essentially identical to the initial full model. For the LAI prediction
Statistical performance of the models for LAI prediction. Models were compared
model, the RFE only dropped the Sentinel-2 band “B02” (blue band).
by fold, which gives the average performance over all k folds from cross-vali-
dation (CV). Models were also compared by their global performance which is
The combination of FFS and random cross-validation selected latitude,
the RMSE or R2 over every data point predicted during cross-validation. Bold longitude, DEM and aspect for LULC classification and DEM, longitude,
numbers indicate a spatial validation that must be considered as the valid latitude, B12, B07, aspect and slope for LAI predictions, in decreasing
performance for the prediction task. For an overview on the model-ID see also order of importance.
Fig. 4. FFS with spatial cross-validation identified the geographic co-
ordinates and elevation as irrelevant or even counterproductive and
ID Variables CV By fold Global
dropped them from the models. The final model used only a subset of
RMSE R2
RMSE R2 the spectral variables and the slope for the LULC classification. Here,
green, blue, red and the standard deviation of the pca in a 9 × 9 en-
1a All Random 0.96 0.87 0.97 0.86
vironment made the largest contributions (Fig. 8a). For LAI predictions,
1b All Spatial 1.25 0.07 1.75 0.58
2a Selected by RFE “spatial” Random 0.93 0.88 0.95 0.87 only the bands B05, B07 (both red edge), B03 (green) and B8A (narrow
2b Selected by RFE “spatial” Spatial 1.22 0.06 1.73 0.58 NIR) were identified as important. However, B03 and B8A only slightly
3a Selected by FFS random Random 0.91 0.88 0.92 0.88 decreased the RMSE compared to the model that used B05 and B07 only
3b Selected by FFS random Spatial 1.23 0.04 1.80 0.58
(Fig. 8b).
4a Selected by FFS spatial Random 1.12 0.83 1.14 0.81
4b Selected by FFS spatial Spatial 1.20 0.06 1.64 0.63
3.3. Visual assessment of the spatial prediction

and random FFS, RMSE = 0.93 for RFE and 0.91 for FFS random). The model that used all variables to predict LULC led to noticeably
Hence, neither a FFS with random selection nor the RFE approach does linear features when making spatial predictions for the full study area
prevent spatial overfitting even though spatial folds have been used for (Fig. 9 no selection; the RFE-based model produces a quasi identical

Fig. 6. Comparison between observed and predicted LAI va-

lues based on random (a) and spatial (b) cross-validation.
Colors indicate the individual 11 folds. Note that the resulting
models are quasi identical regardless of the validation strategy
being used since cross-validation in this case serves mtry
tuning and validation purposes only. (For interpretation of the
references to color in the text, the reader is referred to the web
version of this article.)

6
H. Meyer, et al. Ecological Modelling 411 (2019) 108815

Fig. 7. Relative scaled importance of the predictor variables within the Random Forest models using all variables or using an RFE approach for the case study of
predicting (a) LULC and (b) LAI. See Sections 2.1.2 and 2.2.2 for further explanations of the variables.

spatial prediction). A clear linear delineation was made between beech removed spectral variables from the model.
forest and grassland that does not correspond to the visual inspection of When variables were selected by spatial FFS, the coordinates and
the underlying aerial image (Fig. 9 RGB). An obvious patch of forest in elevation were removed by the algorithm and the classification was
the southeastern part of the image was classified as grassland. Field and based on the spectral variables (Fig. 9 spatial selection). The result
road were clearly confused for one another in the northwestern corner. showed much greater local variability that was clearly driven by the
A round patch of Douglas fir in the southwestern quarter of the image underlying spectral information rather than gradual changes driven by
can be clearly associated to the highest elevation of the forest. Elevation geolocation. No linear artefacts were observed.
appeared to be an important factor in the prediction of water, since Similar though less striking patterns were found for the LAI pre-
parts of fields in the north of the image corresponding to the lowest dictions. Using all potential predictor variables led to a visible linear
elevations (Fig. 2) were falsely classified as such. These areas were feature dividing generally lower LAI values to the east from generally
visually distinguishable from water in the RGB. Several other patterns higher values to the west (Fig. 10 no selection). Such features became
that did not correspond to a visual interpretation of the RGB were also more obvious when FFS with random cross-validation was applied
present. (Fig. 10 random selection). When FFS selected variables with spatial
Random variable selection (FFS with random selection) enhanced cross-validation, no geolocation variables were selected and the results
the problem and linear features became more obvious (Fig. 9 random showed no obvious artefacts in the spatial prediction (Fig. 10 spatial
selection). The prediction has an overall smooth appearance as FFS selection).

Fig. 8. Performance of FFS with variables selected by

spatial cross-validation for prediction of (a) LULC and
(b) LAI. The first point indicates the model's perfor-
mance using the two variables that lead to the best
spatial performance. Subsequent points indicate the
model's performance with the addition of the next best
variable, i.e. the third point represents the top four
variables. Bars represent the standard deviation over
the k spatial folds. See Sections 2.1.2 and 2.2.2 for
further explanations of the variables.

7
H. Meyer, et al. Ecological Modelling 411 (2019) 108815

Fig. 9. RGB representation of the study area on the basis of the aerial image (RGB), spatial LULC predictions by the model that used all potential predictor variables
(no selection), the model with variables being selected by random FFS (random selection), as well as spatial FFS (spatial selection).

4. Discussion Standard validation procedures that use random subsets of the dataset
(i.e. random k-fold cross-validation) produce overoptimistic estimates
4.1. Importance of spatial validation about the model performance (in this case “nearly perfect classifica-
tion” of LULC and low errors for LAI predictions). These do not provide
The results clearly highlight again the necessity of spatial validation information about the actual model performance with respect to the
for realistically assessing the performance of spatial prediction models. prediction of any other place than the sampling locations (i.e. create a

Fig. 10. RGB representation of the study area on the basis of the Sentinel-2 image (RGB), spatial LAI predictions by the model that used all potential predictor
variables (no selection), the model with variables being selected by random FFS (random selection), as well as spatial FFS (spatial selection).

8
H. Meyer, et al. Ecological Modelling 411 (2019) 108815

map of LULC or LAI). Predicting any and not just the training locations, variables, those that are misleading and responsible for overfitting are
however, is the aim of most spatial prediction models, and the perfor- often highly ranked (e.g. latitude, longitude in this study) and hence not
mance estimates must account for that which is increasingly and con- removed by RFE.
sistently recommended in recent literature (e.g. Wenger and Olden, This study automatically analyzes which variables are misleading,
2012; Juel et al., 2015; Roberts et al., 2017; Valavi et al., 2018; counterproductive and cause overfitting and hence must be removed
Pohjankukka et al., 2017; Cánovas-García et al., 2017). Random vali- from the models. We show that removing these variables from the
dation is not a meaningful strategy for spatial prediction tasks. This is models improves the statistical spatial performance; a visual inspection
especially essential when flexible algorithms are applied to highly also confirmed reliable patterns compared to the common model that
clustered training samples that cause the risk of overfitting caused by uses all variables. The increase in statistical performance was less ob-
the spatial autocorrelation of the predictor and response variables. vious than in Meyer et al. (2018), who used spatio-temporal data that
can be explained by a stronger autocorrelation due to the application to
4.2. Relevance of spatial variable selection long time series. Therefore, improvements in performance will likely
track with increasing degrees of autocorrelation if the sampling design
It is not surprising that the model that uses all variable showed a includes clear spatial clusters.
strong performance in random cross-validation as compared to spatial The results of this study strongly suggest that spatial cross-valida-
cross-validation considering that within the polygons of digitized LULC tion needs to be considered not only for model validation and model
or lidar-derived LAI training sites, the samples feature similar proper- tuning (see Schratz et al., 2019, for a study on the relevance of spatial
ties in their predictor variable space. Hence large parts of the training validation for hyperparameter tuning in machine learning applications)
samples are not independent from one another. This leads to overfitting but also for variable selection, hence during all steps of model building.
and incorrectly assigning high importance to the variables that re-
present the spatial location, which is especially clear for geolocation 4.3. Need for visual assessment in addition to statistical validation
variables (i.e. latitude, longitude). Therefore, many studies have un-
surprisingly identified coordinates as one of the, if not the, most im- The results also show that statistical validation alone is insufficient
portant predictor, such as for tree species distribution (Attorre et al., to validate spatial prediction models. Both the model using all potential
2011), monthly precipitation (Jing et al., 2016), deforestation (Zanella predictors and the one using spatially selected variables perform sta-
et al., 2017), phytoplankton abundance (Roubeix et al., 2016) and ex- tistically similar in spatial cross-validation. Hence, we conclude that the
plaining the spatial variability of soil organic carbon (Yang et al., models perform equally well, statistically speaking. A visual assessment
2016). According to our results, spatial variable selection would have reveals that this assumption does not hold true, however. Removing
very likely removed geolocation variables from these studies’ models. In misleading variables dramatically changes the actual outcome as pat-
addition to geolocation variables such as coordinates or Euclidean terns in the LULC and LAI predictions are considerably different. Again,
distance fields (Behrens et al., 2018), variables that are unique to a this highlights the need for spatial variable selection. In several other
certain spatial cluster would be problematic. For example, Meyer et al. studies, artefacts are mainly visible as clear linear features and can most
(2016) show that elevation can complicate models when it is clearly certainly be traced back to the geolocation variables. Jing et al. (2016)
indicative of one spatial cluster – in this study, circular patterns of and Shi et al. (2015) used coordinates for downscaling precipitation,
Douglas fir were predicted for the areas of the forest with the greatest which the Random Forest algorithm identified as the most important
elevation, which is a clear artefact caused by misinterpretation of ele- variable, but which resulted in visible linear patterns in the spatial
vation as a predictor variable. Spatial FFS removed elevation as it was prediction. Mud content prediction by Li et al. (2011) also shows linear
identified as unimportant or counterproductive as evinced by improved patterns that are most likely caused by the inclusion of latitude and
visual and statistical results. longitude as predictors. In a study by Fox et al. (2017) the Random
We show that including location variables does not solve the pro- Forest algorithm also ranked latitude and longitude as important, yet
blem of autocorrelation but intensifies the problem, at least for spatially the resulting marine bird distribution along the Canadian coast shows
clustered data. This finding contrasts with recommendations from clear linear cuts. These examples visually highlight the issues that in-
previous studies (Mascaro et al., 2013; Cracknell and Reading, 2014). cluding geolocation variables can cause. They also underline the im-
Though Random Forests are known for being robust to uninformative portance of spatial cross-validation for spatial error assessment in
predictor variables, this study clearly shows that misleading variables conjunction with spatial variable selection to ensure that only variables
can have negative effects on the models. This notion is also supported with actual predictive power beyond the training locations are in-
by Rocha et al. (2018), who showed that spatial predictions of plant cluded.
traits suffer when models include spatial relations. The phenomenon,
however, can only be detected if spatial cross-validation is used. Spatial 5. Conclusions
models evaluated in a default random way will still feign a high spatial
performance. This study underlines the necessity of spatial validation strategies in
Spatial cross-validation provides reliable performance measures for spatial machine learning applications. Results will likely be over-
spatial models. However, it does not change the model itself. During the optimistic if these strategies are ignored. This is especially the problem
internal Random Forest training, variables are not selected by spatial when there is strong spatial autocorrelation in the data and when
cross-validation but by the out-of-bag (oob) error which is based on training samples are clustered in space.
random bootstrapping. Hence in a Random Forest training, variables However, spatial machine learning applications should not be re-
are not selected by their spatial importance. A spatial variable selection stricted to the usage of spatial validation. This study shows that certain
is therefore required to remove misleading variables from the models. variables are responsible for overfitting that causes strong random
Using geolocation variables, the algorithm could reproduce training performance but a failure in predicting any other than the training
samples with highest performance but only a selection of spatially samples location. This is especially evident for geolocation variables
meaningful variables allowed for predictions beyond the locations of (e.g. latitude, longitude). When such variables are used in spatial
the training samples. modelling where training samples are highly clustered in space, they
Having a look at the internal variable importance ranking of the lead the algorithms to effectively reproduce the training data but lead
Random Forests algorithm (Fig. 7) also explains why recursive feature the model to fail predicting on new samples. Hence, the applied
selection cannot help, even if spatial cross-validation determines the Random Forest algorithm cannot interpret such variables in a mean-
optimal variables: Since RFE is based on the importance ranking of the ingful way. Spatial variable selection is required to automatically select

9
H. Meyer, et al. Ecological Modelling 411 (2019) 108815

variables that are useful in a Random Forest setup, for example the 2013.08.011.
suggested forward feature selection that selects variables according to Gudmundsson, L., Seneviratne, S.I., 2015. Towards observation-based gridded runoff
estimates for Europe. Hydrol. Earth Syst. Sci. 19, 2859–2879.
their contribution for spatial predictions. Spatial validation should Hengl, T., Nussbaum, M., Wright, M., Heuvelink, G., Gräler, B., 2018. Random forest as a
hence be considered during all steps of modelling, from hyperparameter generic framework for predictive modeling of spatial and spatio-temporal variables.
tuning, variable selection to performance estimation. PeerJ Preprints. https://doi.org/10.7287/peerj.preprints.26693v3.
Hessische Verwaltung für Bodenmanagement und Geoinformation, 2018a. Aerial
Like most other machine learning algorithms, Random Forests have Imagery.
the reputation that no assumptions about the data distribution are ne- Hessische Verwaltung für Bodenmanagement und Geoinformation, 2018b. Lidar data.
cessary. However, the results of this study show that it might be ne- Hunt, E.R., Doraiswamy, P.C., McMurtrey, J.E., Daughtry, C.S., Perry, E.M., Akhmedov,
B., 2013. A visible band index for remote sensing leaf chlorophyll content at the
cessary to revisit this idea and general guidelines should be formulated canopy scale. Int. J. Appl. Earth Obs. Geoinf. 21, 103–112. https://doi.org/10.1016/
to make applications more objective. j.jag.2012.07.020. http://www.sciencedirect.com/science/article/pii/
Finally, the results of this study allow the conclusion that ignoring S0303243412001791.
Janatian, N., Sadeghi, M., Sanaeinejad, S.H., Bakhshian, E., Farid, A., Hasheminia, S.M.,
spatial dependencies in machine learning applications for spatial pre-
Ghazanfari, S., 2017. A statistical framework for estimating air temperature using
dictions carries a high risk of developing models that can reproduce MODIS land surface temperature data. Int. J. Climatol. 37, 1181–1194. https://doi.
training data well but do not make reliable spatial predictions. Reliable org/10.1002/joc.4766.
spatial predictions can only be achieved if spatial dependencies are Jing, W., Yang, Y., Yue, X., Zhao, X., 2016. A comparison of different regression algo-
rithms for downscaling monthly satellite-based precipitation over North China.
taken into account during the modelling process, i.e. not only for the Remote Sens. 8, 835. https://doi.org/10.3390/rs8100835.
purpose of model validation, but also for the selection of appropriate Juel, A., Groom, G.B., Svenning, J.-C., Ejrnæs, R., 2015. Spatial application of Random
predictor variables. Forest models for fine-scale coastal vegetation classification using object based
analysis of aerial orthophoto and DEM data. Int. J. Appl. Earth Obs. Geoinf. 42,
106–114. https://doi.org/10.1016/j.jag.2015.05.008. http://www.sciencedirect.
Acknowledgments com/science/article/pii/S0303243415001178.
Kuhn, M., 2016. caret: Classification and Regression Training. R package version 6.0-68.
https://CRAN.R-project.org/package=caret.
This work was conducted within the Natur 4.0 | Sensing Kuhn, M., Johnson, K., 2013. Applied Predictive Modeling, 1st ed. Springer, New York.
Biodiversity project funded by the Hessian state offensive for the de- Langella, G., Basile, A., Bonfante, A., Terribile, F., 2010. High-resolution space-time
velopment of scientific-economic excellence (LOEWE). rainfall analysis using integrated ANN inference systems. J. Hydrol. 387, 328–342.
Lary, D.J., Alavi, A.H., Gandomi, A.H., Walker, A.L., 2016. Machine learning in geos-
ciences and remote sensing. Geosci. Front. 7, 3–10. https://doi.org/10.1016/j.gsf.
References 2015.07.003.
Le Rest, K., Pinaud, D., Monestiez, P., Chadoeuf, J., Bretagnolle, V., 2014. Spatial leave-
one-out cross-validation for variable selection in the presence of spatial auto-
Appelhans, T., Mwangomo, E., Hardy, D.R., Hemp, A., Nauss, T., 2015. Evaluating ma-
correlation. Global Ecol. Biogeogr. 23, 811–820. https://doi.org/10.1111/geb.
chine learning approaches for the interpolation of monthly air temperature at Mt.
12161.
Kilimanjaro, Tanzania. Spat. Stat. 14 (Part A), 91–113. https://doi.org/10.1016/j.
Li, J., Heap, A.D., Potter, A., Daniell, J.J., 2011. Application of machine learning methods
spasta.2015.05.008.
to spatial interpolation of environmental variables. Environ. Model. Softw. 26,
Attorre, F., Alfò, M., Sanctis, M.D., Francesconi, F., Valenti, R., Vitale, M., Bruno, F., 2011.
1647–1659.
Evaluating the effects of climate change on tree species abundance and distribution in
Liaw, A., Wiener, M., 2002. Classification and regression by Random Forest. R News 2,
the Italian peninsula. Appl. Veg. Sci. 14, 242–255. http://www.jstor.org/stable/
18–22.
41058163.
Mascaro, J., Asner, G.P., Knapp, D.E., Kennedy-Bowdoin, T., Martin, R.E., Anderson, C.,
Bahn, V., McGill, B.J., 2013. Testing the predictive performance of distribution models.
Higgins, M., Chadwick, K.D., 2013. A tale of two “Forests”: random forest machine
Oikos 122, 321–331. https://doi.org/10.1111/j.1600-0706.2012.00299.x.
learning aids tropical forest carbon mapping. PLOS ONE 9, e85993. http://www.
Behrens, T., Schmidt, K., Viscarra Rossel, R.A., Gries, P., Scholten, T., MacMillan, R.A.,
ncbi.nlm.nih.gov/pmc/articles/PMC3904849/.
2018. Spatial modelling with euclidean distance fields and machine learning. Eur. J.
Meyer, H., 2018. CAST: ‘caret’ Applications for Spatial-Temporal Models. R Package
Soil Sci. 69, 757–770. https://doi.org/10.1111/ejss.12687.
Version 0.2.1. https://CRAN.R-project.org/package=CAST.
Breiman, L., 2001. Random forests. Mach. Learn. 45, 5–32. https://doi.org/10.1023/
Meyer, H., Katurji, M., Appelhans, T., Müller, M.U., Nauss, T., Roudier, P., Zawar-Reza,
A:1010933404324.
P., 2016. Mapping daily air temperature for Antarctica based on MODIS LST. Remote
Brenning, A., 2005. Spatial prediction models for landslide hazards: review, comparison
Sens. 8, 732. https://doi.org/10.3390/rs8090732.
and evaluation. Nat. Hazards Earth Syst. Sci. 5, 853–862. https://doi.org/10.5194/
Meyer, H., Kühnlein, M., Reudenbach, C., Nauss, T., 2017a. Revealing the potential of
nhess-5-853-2005.
spectral and textural predictor variables in a neural network-based rainfall retrieval
Brungard, C.W., Boettinger, J.L., Duniway, M.C., Wills, S.A., T.C.E Jr., , 2015. Machine
technique. Remote Sens. Lett. 8, 647–656. https://doi.org/10.1080/2150704X.2017.
learning for predicting soil classes in three semi-arid landscapes. Geoderma 239–240,
1312026.
68–83. https://doi.org/10.1016/j.geoderma.2014.09.019.
Meyer, H., Lehnert, L.W., Wang, Y., Reudenbach, C., Nauss, T., Bendix, J., 2017b. From
Cánovas-García, F., Alonso-Sarría, F., Gomariz-Castillo, F., nate Valdivieso, F.O., 2017.
local spectral measurements to maps of vegetation cover and biomass on the Qinghai-
Modification of the random forest algorithm to avoid statistical dependence problems
Tibet-Plateau: do we need hyperspectral information? Int. J. Appl. Earth Obs. Geoinf.
when classifying remote sensing imagery. Comput. Geosci. 103, 1–11. https://doi.
55, 21–31. https://doi.org/10.1016/j.jag.2016.10.001.
org/10.1016/j.cageo.2017.02.012. http://www.sciencedirect.com/science/article/
Meyer, H., Reudenbach, C., Hengl, T., Katurji, M., Nauss, T., 2018. Improving perfor-
pii/S0098300416303909.
mance of spatio-temporal machine learning models using forward feature selection
Cohen, J., 1960. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20,
and target-oriented validation. Environ. Model. Softw. 101, 1–9. https://doi.org/10.
37–46. https://doi.org/10.1177/001316446002000104.
1016/j.envsoft.2017.12.001.
Cracknell, M.J., Reading, A.M., 2014. Geological mapping using remote sensing data: a
Micheletti, N., Foresti, L., Robert, S., Leuenberger, M., Pedrazzini, A., Jaboyedoff, M.,
comparison of five machine learning algorithms, their response to variations in the
Kanevski, M., 2014. Machine learning feature selection methods for landslide sus-
spatial distribution of training data and the use of explicit spatial information.
ceptibility mapping. Math. Geosci. 46, 33–57.
Comput. Geosci. 63, 22–33. https://doi.org/10.1016/j.cageo.2013.10.008.
Planetary Habitability Laboratory, 2015. Visible Vegetation Index (vvi). http://phl.upr.
Fox, C.H., Huettmann, F.H., Harvey, G.K.A., Morgan, K.H., Robinson, J., Williams, R.,
edu/projects/visible-vegetation-index-vvi.
Paquet, P.C., 2017. Predictions from machine learning ensembles: marine bird dis-
Pohjankukka, J., Pahikkala, T., Nevalainen, P., Heikkonen, J., 2017. Estimating the
tribution and density on Canada's pacific coast. Mar. Ecol. Prog. Ser. 566, 199–216.
prediction performance of spatial models via spatial k-fold cross validation. Int. J.
https://www.int-res.com/abstracts/meps/v566/p199-216/.
Geogr. Inform. Sci. 31, 2001–2019. https://doi.org/10.1080/13658816.2017.
Gasch, C.K., Hengl, T., Gräler, B., Meyer, H., Magney, T.S., Brown, D.J., 2015. Spatio-
1346255.
temporal interpolation of soil water, temperature, and electrical conductivity in
R Core Team, 2018. R: A Language and Environment for Statistical Computing. R
3D + T: the cook agronomy farm data set. Spat. Stat. 14 (Part A), 70–90.
Foundation for Statistical Computing Vienna, Austria. https://www.R-project.org/.
Georganos, S., Grippa, T., Gadiaga, A.N., Linard, C., Lennert, M., Vanhuysse, S., Mboga,
Roberts, D.R., Bahn, V., Ciuti, S., Boyce, M.S., Elith, J., Guillera-Arroita, G., Hauenstein,
N., Wolff, E., Kalogirou, S., 2019. Geographical random forests: a spatial extension of
S., Lahoz-Monfort, J.J., Schröder, B., Thuiller, W., Warton, D.I., Wintle, B.A., Hartig,
the random forest algorithm to address spatial heterogeneity in remote sensing and
F., Dormann, C.F., 2017. Cross-validation strategies for data with temporal, spatial,
population modelling. Geocarto Int. 1–16. https://doi.org/10.1080/10106049.2019.
hierarchical, or phylogenetic structure. Ecography. https://doi.org/10.1111/ecog.
1595177.
02881.
Getzin, S., Fischer, R., Knapp, N., Huth, A., 2017. Using airborne lidar to assess spatial
Rocha, A.D., Groen, T.A., Skidmore, A.K., Darvishzadeh, R., Willemen, L., 2018. Machine
heterogeneity in forest structure on mount Kilimanjaro. Landsc. Ecol. 32, 1881–1894.
learning using hyperspectral data inaccurately predicts plant traits under spatial
https://doi.org/10.1007/s10980-017-0550-7.
dependency. Remote Sens. 10https://doi.org/10.3390/rs10081263. http://www.
Ghosh, A., Joshi, P., 2014. A comparison of selected classification algorithms for mapping
mdpi.com/2072-4292/10/8/1263.
bamboo patches in lower gangetic plains using very high resolution WorldView 2
Roubeix, V., Danis, P.-A., Feret, T., Baudoin, J.-M., 2016. Identification of ecological
imagery. Int. J. Appl. Earth Obs. Geoinf. 26, 298–311. https://doi.org/10.1016/j.jag.
thresholds from variations in phytoplankton communities among lakes: contribution

10
H. Meyer, et al. Ecological Modelling 411 (2019) 108815

to the definition of environmental standards. Environ. Monit. Assess. 188, 246. ONE 12, 1–18. https://doi.org/10.1371/journal.pone.0179473.
https://doi.org/10.1007/s10661-016-5238-y. Wang, Y., Wu, G., Deng, L., Tang, Z., Wang, K., Sun, W., Shangguan, Z., 2017. Prediction
Schratz, P., Muenchow, J., Iturritxa, E., Richter, J., Brenning, A., 2019. Hyperparameter of aboveground grassland biomass on the Loess Plateau, China, using a random forest
tuning and performance assessment of statistical and machine-learning algorithms algorithm. Sci. Rep. 7, 6940. https://doi.org/10.1038/s41598-017-07197-6.
using spatial data. Ecol. Model. 406, 109–120. https://doi.org/10.1016/j.ecolmodel. Wenger, S.J., Olden, J.D., 2012. Assessing transferability of ecological models: an un-
2019.06.002. http://www.sciencedirect.com/science/article/pii/ derappreciated aspect of statistical validation. Methods Ecol. Evol. 3, 260–267.
S0304380019302145. https://doi.org/10.1111/j.2041-210X.2011.00170.x.
Shi, Y., Song, L., Xia, Z., Lin, Y., Myneni, R.B., Choi, S., Wang, L., Ni, X., Lao, C., Yang, F., Xie, Y., Eftelioglu, E., Ali, R.Y., Tang, X., Li, Y., Doshi, R., Shekhar, S., 2017.
2015. Mapping annual precipitation across mainland China in the period 2001–2010 Transdisciplinary foundations of geospatial data science. ISPRS Int. J. Geo-Inform.
from TRMM3B43 product using spatial downscaling approach. Remote Sens. 7, 6https://doi.org/10.3390/ijgi6120395. http://www.mdpi.com/2220-9964/6/12/
5849–5878. https://doi.org/10.3390/rs70505849. 395.
Stevens, A., Nocita, M., Tóth, G., Montanarella, L., van Wesemael, B., 2013. Prediction of Yang, R.-M., Zhang, G.-L., Yang, F., Zhi, J.-J., Yang, F., Liu, F., Zhao, Y.-G., Li, D.-C., 2016.
soil organic carbon at the European scale by visible and near infraRed reflectance Precise estimation of soil organic carbon stocks in the northeast Tibetan plateau. Sci.
spectroscopy. PLOS ONE 8, 1–13. https://doi.org/10.1371/journal.pone.0066409. Rep. 6, 21842. https://doi.org/10.1038/srep21842.
Valavi, R., Elith, J., Lahoz-Monfort, J.J., Guillera-Arroita, G., 2018. blockcv: an r package Zanella, L., Folkard, A.M., Blackburn, G.A., Carvalho, L.M.T., 2017. How well does
for generating spatially or environmentally separated folds for k-fold cross-validation random forest analysis model deforestation and forest fragmentation in the Brazilian
of species distribution models. BioRxiv. https://doi.org/10.1101/357798. Atlantic forest? Environ. Ecol. Stat. 24, 529–549. https://doi.org/10.1007/s10651-
Walsh, E.S., Kreakie, B.J., Cantwell, M.G., Nacci, D., 2017. A random forest approach to 017-0389-8.
predict the spatial distribution of sediment pollution in an estuarine system. PLOS

11