Robust global sensitivity analysis under deep uncertainty via scenario analysis

Environmental Modelling & Software 76 (2016) 154e166 Contents lists available at ScienceDirect Environmental Modelling & Software journal homepage: www.elsevier.com/locate/envsoft Robust global sensitivity analysis under deep uncertainty via scenario analysis Lei Gao a, *, Brett A. Bryan a, Martin Nolan a, Jeffery D. Connor a, Xiaodong Song b, Gang Zhao c a CSIRO Land and Water, Private Mail Bag 2, Glen Osmond, SA 5064, Australia b Institute of Remote Sensing and Information Application, Zhejiang University, Hangzhou, 310058, China c Crop Science Group, Institute of Crop Science and Resource Conservation (INRES), University of Bonn, Katzenburgweg 5, D-53115 Bonn, Germany a r t i c l e i n f o a b s t r a c t Article history: Complex social-ecological systems models typically need to consider deeply uncertain long run future Received 20 March 2015 conditions. The influence of this deep (i.e. incalculable, uncontrollable) uncertainty on model parameter Received in revised form sensitivities needs to be understood and robustly quantified to reliably inform investment in data 30 October 2015 collection and model refinement. Using a variance-based global sensitivity analysis method (eFAST), we Accepted 7 November 2015 Available online 21 November 2015 produced comprehensive model diagnostics of a complex social-ecological systems model under deep uncertainty characterised by four global change scenarios. The uncertainty of the outputs, and the in- fluence of input parameters differed substantially between scenarios. We then developed sensitivity Keywords: Global sensitivity analysis indicators that were robust to this deep uncertainty using four criteria from decision theory. The pro- Robust sensitivity analysis posed methods can increase our understanding of the effects of deep uncertainty on output uncertainty eFAST and parameter sensitivity, and incorporate the decision maker's risk preference into modelling-related Decision theory activities to obtain greater resilience of decisions to surprise. Land use change © 2015 Elsevier Ltd. All rights reserved. Deep uncertainty 1. Introduction drivers, coupled with specific natural, economic, and social trends (Millennium Ecosystem Assessment, 2005a; IPCC, 2007; Moss et al., Large, integrated models of complex social-ecological systems 2010). Often, probabilities of the occurrence of these future states are increasingly used to assess the impacts of land use changes and are not known (or not agreed on by experts) and predictions based ecosystem services under key influences: future climates, popula- on past data are not reliable. This kind of uncertainty is incalculable tion dynamics, resource scarcity, as well as demand and supply in and uncontrollable, and characterised as deep uncertainty (Knight, local and global markets for food, energy, carbon and other com- 1921; Ben-Haim, 2001; Groves and Lempert, 2007; Walker et al., modities (Schaldach et al., 2011; Bateman et al., 2013; Connor et al., 2010; Wintle et al., 2010; Cox, 2012). Deep uncertainty is distinct 2015). Global sensitivity analysis (GSA) is increasingly being used to from ‘probabilistic uncertainty’ (Morgan and Henrion, 1992) or analyse these modelsdidentifying critical model input parameters ‘stochastic uncertainty’ (Quade, 1989) which include frequency- and quantifying the impact of uncertainty in these parameters on based probabilities or subjective (Bayesian) probabilities. Typical model outputs (Saltelli et al., 2008). Knowing influential factors can sources of deep uncertainty are the future state of the world help to prioritise data collection, identify non-influential model (objective) and the decision maker (subjective). The presence of parameters, isolate major parametric uncertainty sources, under- deep uncertainty challenges global sensitivity analyses in two main stand model structure, and check model errors. Sensitivity analysis ways. First, we need to understand the influence of deep uncer- is often performed on simple models fit to observed data. However, tainty on model parameter sensitivity. Second, we need to quantify complex social-ecological systems are typically subject to uncertain sensitivity in a way that is robust to deep uncertainty and can be future conditions, which are usually shaped by local and global used to reliably inform investment in gathering and analysing additional data and refining model input parameter estimates. As traditional predict-then-act approaches are often inadequate * Corresponding author. CSIRO Land and Water, Private Mail Bag 2, Waite Road, for social-ecological assessments when faced with uncontrollable Glen Osmond, SA 5064, Australia and deep uncertainty (Peterson et al., 2003; Popper et al., 2005), a E-mail address: [email protected] (L. Gao). http://dx.doi.org/10.1016/j.envsoft.2015.11.001 1364-8152/© 2015 Elsevier Ltd. All rights reserved.  L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 155 common approach is to identify a set of plausible, internally- practically-feasible strategies to make optimal decisions under consistent future scenarios covering a range of possible condi- uncertainty (Morgan et al., 1992). In situations of deep uncertainty, tions (IPCC, 2000; Millennium Ecosystem Assessment, 2005b). a combination of scenario planning and decision theory can be a When the future is inherently uncertain, using scenario planning useful way to expand the space of possible states and associated rather than a detailed prediction of the most likely future can help outcomes, and to make decisions given current understanding. A decision makers recognize and respond more effectively to changes strategy in decision theory usually characterises a decision (in our (Mahmoud et al., 2009; Wilkinson and Kupers, 2013; Kirby et al., case, quantifying parameter sensitivities) as a function of different 2014). Recent influential efforts using scenarios to address deep possible states of the world (scenarios), alternatives/choices, utility uncertainty include the Intergovernmental Panel on Climate functions between scenarios and alternatives, and a decision cri- Change (IPCC)’s representative concentration pathways (Moss et al., terion. Then, a good decision is made in situations of deep uncer- 2010; van Vuuren et al., 2011) and the Millennium Ecosystem tainty by incorporating the decision maker's risk preference over Assessment (Millennium Ecosystem Assessment, 2005a). Although different outcomes in the decision criterion. The robustness of this scenarios enable a creative, participatory, and systematic approach decision is provided by the consideration of multiple possible fu- to exploring potential futures (Peterson et al., 2003; Mahmoud tures with acceptable levels of satisfaction and risk. et al., 2009; Polasky et al., 2011; Kirby et al., 2015), their weak- We undertook a GSA for a complex social-ecological systems ness lies in the reliance on a manageably small number of scenarios modeldthe Australian continental Land Use Trade-Offs (LUTO) to capture the full range of future uncertainty (Bryant and Lempert, model (Bryan et al., 2014; Connor et al., 2015; Bryan et al., in 2010). In response, recent work in bottom-up decision frameworks review), and assessed the impact of scenario choice on parameter (Groves and Lempert, 2007; Bryant and Lempert, 2010; Kasprzyk sensitivities. LUTO provides an integrated, evidence-based assess- et al., 2013; Matrosov et al., 2013; Gao et al., 2014; Kwakkel et al., ment of a range of possible outlooks for Australian land use and 2014; Herman et al., 2015) has advocated the discovery of many ecosystem services (Bryan et al., 2014; Connor et al., 2015). The scenarios by applying statistical or data-mining algorithms. Sce- model uses scenarios combining both global and domestic drivers narios derived both through traditional participatory means, and to explore the implications of uncertain future conditions (Hatfield- more recently through discovery techniques, have been widely Dodds et al., 2015). It is necessary to understand how the scenarios used to provide context for systems models in assessing future influence model parameter sensitivities and to identify a parameter management plans of natural resources (Bohensky et al., 2006; sensitivity metric that is robust under different scenarios. We Bryant and Lempert, 2010; Bryan et al., 2011; Kasprzyk et al., applied eFAST in a GSA of the LUTO model under four global out- 2013; Davies-Barnard et al., 2014). Sensitivity analysis of these looks. We then statistically evaluated the influence of different systems models is essential to understand model structure and scenarios on output uncertainty resulting from the variation in behaviour, uncertainty, and parameter sensitivity. However, vary- input parameters. We identified how the scenarios affected influ- ing individual parametersda requirement of sensitivity ana- ential, non-influential, and interacting parameters in the model. lysisdmay jeopardize the internal consistency of scenarios by We also tested for statistical differences in the variance contribu- combining implausible parameter combinations. tion of input parameters to model outputs under different sce- One way to quantify the influence of deep uncertainty on model narios. Finally, we calculated a set of sensitivity indicators that were sensitivity is by performing sensitivity analysis (SA) under different robust to uncertain future conditions by using four criteria from scenarios. Two broad SA strategies are commonly used to explore decision theory. We discuss the implications for undertaking GSA the parameter space: local SA (LSA) and global SA (GSA). GSA on complex models under conditions of deep uncertainty. overcomes the two main drawbacks of LSA (i.e. exploration only within a small interval around the baseline point, and ignorance of 2. The social-ecological model input parameter interaction effects) by exploring the full multi- variate space of a model simultaneously. Among the several GSA 2.1. Model overview and scenarios methods proposed, variance-based methods (e.g., Sobol', 1993; Saltelli et al., 2010) decompose the variance of a model output The LUTO model identifies potential land use futures for Aus- into fractions that can be attributed to model inputs. They have tralia's intensively managed agricultural landdan area of 85.3 recently been applied in environmental modelling such as hy- million ha stretching from central and eastern Queensland, through drology (Nossent et al., 2011; Vezzaro and Mikkelsen, 2012; Shin New South Wales, Victoria and South Australia, to the cropping land et al., 2013; Gan et al., 2014), forestry (Miao et al., 2011; Song of south-west Western Australia (Fig. 1). Within the study area, 33 et al., 2012, 2013), agriculture (Confalonieri et al., 2010; Varella million ha are currently used for cattle production, 18 million ha for et al., 2010; DeJonge et al., 2012; Zhao et al., 2014), environ- sheep production, 3 million ha for dairy, and 25 million ha for grain mental chemistry (Annoni et al., 2011), ecology (Makler-Pick et al., production. The model runs on an annual time step over the time  2011), and waste disposal (Freni and Mannina, 2010; Chen et al., period 2013e2050, at a spatial resolution of 1.1 km (0.01 ) grid cells. 2012; Cosenza et al., 2013). In particular, the extended Fourier It models economic profitability and competition for land between Amplitude Sensitivity Testing (eFAST) has attracted attention five main land use classes: agriculture, carbon plantings, environ- (Marino et al., 2008; Chen et al., 2012; Cosenza et al., 2013; Wang mental plantings, biofuels, and bioenergy. The LUTO model in- et al., 2013; Vanuytrecht et al., 2014; Zhao et al., 2014) due to its tegrates a range of environmental and economic data layers and computational efficiency (Zhao et al., 2014) and ability to calculate models of the production, price, and costs of each land use over the total sensitivity index (Saltelli et al., 1999). time, as well as the impacts of each land use for a range of Determining a sensitivity metric/measure that can reliably ecosystem services including carbon sequestration, biodiversity, identify parameter sensitivities and thereby inform data collection food production, water, and energy. Then by aggregating the land and parameter refinement effort under deeply uncertain future use decisions made in each grid cell, the LUTO model identifies conditions can be framed as a problem of making a robust decision potential land use changes over time and the resultant ecosystem (Polasky et al., 2011)d“the best possible choice that one found by service impacts of land use futures. Details of the LUTO model are eliminating all the uncertainty possible within available resources, and described in Connor et al. (2015) and Bryan et al. (2014). then choosing, with known and acceptable levels of satisfaction and Global context for the LUTO model was provided by four global risk” (Ullman, 2006). Decision theory offers scientifically-sound and outlooks (termed L1, M3, M2, and H3)ddefined as plausible, 156 L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 sensitivity and uncertainty analysis of the LUTO model (Table 2). For spatial input parameters, we used a single factor approach such that all grid cells were varied by this factor. Similarly, a multiplier was also applied to time-series values in the scenario parameter group. Because of the absence of information about the prior probability distributions for each parameter, a uniform distribution was assumed. For parameters lacking information determining its range, we set bounds of 30% each side of the reference value following Song et al. (2012) and Van Oijen et al. (2005). We analysed the sensitivity and uncertainty of 24 model outputs (as shown in Table 3) in 2050 from the LUTO simulation to variation in each of the input parameters (Table 2). 3. Methods We conducted a number of experiments to determine whether the uncertainty and sensitivity analysis results were different depending on scenario, and to build a single set of indicators that are robust to future uncertainty. Fig. 1. The study area in Australia. 3.1. Global sensitivity analysis using eFAST internally consistent scenarios for the global economy, population, greenhouse gas emissions, and climate (Hatfield-Dodds et al., eFAST is a variance decomposition method, which uses a peri- 2015). These scenarios were developed based on three cumula- odic sampling method and a Fourier transformation to partition the tive greenhouse emissions trajectories and three UN population whole variance of the model output and quantify the degree to projections. They are not intended to be comprehensive or cover all which variation in each input factor accounts for the output vari- possible future outcomes, and no probability or likelihood is asso- ance. A periodic sampling approach is used to generate a search ciated with them. L1 is characterised by very strong global action on curve in the parameter space and partitioning is implemented by climate change, low global population, and the lowest degree of assigning the periodic sample of each parameter with a distinct climate change. Both M2 and M3 are mid-range climate outlooks frequency. Then a Fourier transformation is applied to the model and differ insofar as M3 has higher population and strong emis- output to measure how strongly a factor's frequency propagates sions abatement effort while M2 has a lower population with from the input to the output, i.e., the variance contribution of the modest abatement effort. H3 involves no emissions abatement, factor to the whole variance of the output (Saltelli et al., 1999; high population, and severe climate change (Table 1). Total demand Marino et al., 2008). The calculation process of first-order and to- for food and energy varies widely due to differences in population tal sensitivity indices using the eFAST method is shown in Sup- and average global income. The modelled grain, carbon, oil, and plementary Material A. livestock price paths under the four scenarios are also summarised We applied eFAST in uncertainty and sensitivity analyses of the in Table 1. LUTO model. LUTO was written in the Python programming lan- guage (van Rossum and The Python Community, 2013). With each model run taking about 40 h, it is relatively computationally 2.2. Input factors and output variables intensive and this poses a computing challenge for GSAs which usually require many model simulations. To address this computing We selected 50 parameters classified into 9 groups for the challenge we used a reduced resolution version (3.3 km grid cells) Table 1 Dimensions of the four global outlooks assessed. Indicator Unit Value in 2010 Scenario L1 M3 M2 H3 Population and economic growth Population in 2050 Billion people 6.9 8.1 10.6 9.3 10.6 1 World GDP per capita in 2050 US$ ‘000 2010 cap 8.8 20.0 18.6 19.3 18.6 World GDP in 2050 US$ ‘000 billion 61.0 161.6 197.0 179.1 197.8 Climate  Temperature increase by 2100 C e 1.3e1.9 2.0e3.0 2.0e3.0 4.0e6.1 Emissions and abatement Benchmark Raditative Concentration Pathway (RCP) e e RCP3-PD RCP4.5 RCP4.5 RCP8.5 Radiative forcing in 2100 Wm 2 e 2.6 4.5 4.5 8.5 Atmospheric emissions in 2100 CO2 ppm e 445 (declining) 650 (stable) 650 (stable) 1360 (rising) Global abatement effort e Very strong Strong Modest No action 1 Emissions per person in 2050 tCO2 cap 7.0 3.1 4.3 5.0 8.7 Coverage of abatement policy e e All sources All, excluding emissions No sources from livestock Projected prices Grain price increase from 2010 to 2050 AUD$ e 75% 118% 11% 61% Carbon price (global) in 2050 AUD$ e 200 119 59 0 Livestock price increase from 2010 to 2050 e 147% 112% 22% 61% Oil price increase from 2010 to 2050 e 42% 44% 45% 43%  L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 157 Table 2 Input parameters of the LUTO model used in the sensitivity and uncertainty analyses. Parameter group Parameter Description Unit Spatial layer Value range Min Max Scenario CropPricePath Crops and horticultural price path n/a N LivestockPricePath Livestock price path n/a N CarbonPricePath Carbon price path $ tCO2 1 N OilPricePath Oil price path n/a N 1 ElectricityPricePath Average wholesale electricity price AUD MWh N AgriculturalProductivity Productivity increases for crops, beef, n/a N 0 3 sheep, dairy and horticulture TreeGrowthProductivity Productivity increases for trees n/a N 0 1 AdoptionHurdleRate Economic adoption hurdle rate. n/a N 1 5 Agricultural returns increased by hurdle rate BiodiversityLevy Amount of levy on carbon plantings % N 0 30 1 BiodiversityFund Initial biodiversity fund amount $ yr N DiscountRate Discount rate n/a N 1 Agriculture profit function CommodityPricePrimary Farm gate price for primary product $ ha Y 1 CommodityPriceSecondary Farm gate price for secondary product $ ha Y (milk and wool) AgriculturalYield Agriculture product quantity produced t ha 1 or DSE ha 1 Y (dry sheep equivalents) LivestockDisposals Livestock disposals DSE ha 1 Y AgAreaCosts Area cost $ ha 1 Y AgQuantityCosts Quantity cost ¼ actual QC* Q $ ha 1 Y AgFixedOperatingCosts Fixed operating cost $ ha 1 Y AgFixedLabourCosts Fixed labour cost $ ha 1 Y AgFixedDepreciationCosts Fixed deprication cost $ ha 1 Y AgWaterCosts Water delivery cost $ ha 1 Y Tree inputs CarbSeqRateCP Carbon plantings average annual CO2 tCO2 ha 1 yr 1 Y sequested over 20 yrs 1 1 CarbSeqRateEP Environmental plantings average tCO2 ha yr Y annual CO2 sequested over 20 yrs RiskFireDroughtCP Percentage of CO2 sequestration to use n/a Y as 100 year accumulated biomass RiskFireDroughtEP Percentage of CO2 sequestration to use n/a Y as 100 year accumulated biomass CarbSeqRiskBuffer Risk buffer for carbon monoculture and n/a N environmental plantings 1 Tree costs EstabCostCP Initial cost of establishing carbon $ ha Y plantings 1 EstabCostEP Initial cost of establishing $ ha Y environmental plantings 1 EstabCostWP Initial cost of establishing biomass $ ha Y plantings 1 1 AnnMangCostCP Annual management costs for carbon $ ha yr N plantings 1 1 AnnMangCostEP Annual management costs for $ ha yr N environmental plantings 1 1 AnnMangCostWP Annual management costs for biomass $ ha yr N plantings 1 HarvestCostWP Harvest cost for biomass plantings. $ ha N Every 10 years. 1 1 Biofuel inputs BiomassGrowthRateWP Woody perennials plantings t ha decade Y 1 WheatStubbleYield Biofuels wheat stubble t ha yr 1 Y 1 WheatGrainYield Biofuels wheat yield (grain) t ha yr 1 Y 1 Biofuel costs WheatPrice Farm gate price for primary product. $ ha Y Price of winter cereals when grain sold for food and stubble for biofuel. WheatAreaCost Area cost $ ha 1 Y WheatQuantityCost Quantity cost ¼ actual QC* Q $ ha 1 Y WheatFixedOperatingCost Fixed operating cost $ ha 1 Y WheatFixedLabourCost Fixed labour cost $ ha 1 Y WheatFixedDepreciationCost Fixed deprication cost $ ha 1 N Climate impacts CCImpactDrylandCrops Climate impacts for dryland crops n/a Y CCImpactDrylandLivestock Climate impacts for dryland crops n/a Y CCImpactTreeGrowth Climate impacts for trees n/a Y 1 1 Water WaterInterceptionTrees Water impact as rate of tree ML ha year Y interception 1 WaterLicensePrice Price of water intercepted by trees $ ML Y CCImpactWaterLicensePrice Cell value of climate impact on water % Y run-off used to adjust water price Others AgPriceElasticityOfDemand Elasticity of demand N 0.8 0.4 BioenergyProfitShare Proportion of difference between petrol N price and biofuel retail price paid to producer 158 L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 Table 3 A list of the major LUTO output variables used in the analyses. Output variable Description Unit ProductionGrains Total amount of grains produced in a year tonne ProductionOtherCrops Total amount of other crops produced in a year Tonne ProductionDairy Total number of dairy cattle head ProductionBeef Total number of beef cattle head ProductionSheep Total number of sheep head CarbonSequestration Creditable carbon sequestration tCO2e WaterIntercepted Annual water interception from trees ML BiodiversityServices Area weighted measure of biodiversity value % AgriculturalProduction Value of agricultural production in 2010 prices $ EnergyBiofuel Total biofuels produced ML EnergyElectricity Total bio-electricity produced GJ AreaGrains Total area of grains ha AreaOtherCrops Total area of other crops ha AreaDairy Total area of dairy ha AreaBeef Total area of beef ha AreaSheep Total area of sheep ha AreaCarbonPlantings Total area of carbon plantings ha AreaEnvironmentalPlantingsC Total area of environmental plantings (carbon price) ha AreaEnvironmentalPlantingsCBF Total area of environmental plantings (carbon price and carbon price þ biodiversity fund) ha AreaWheat (Food þ Biofuel) Total area of crops planted to wheat for food and biofuels ha AreaWheat (Biofuel) Total area of crops planted to wheat for biofuels ha AreaWoodyPerennials (Biofuel) Total area of woody perennials for biofuels ha AreaWheat (Food þ Electricity) Total area of crops planted to wheat for food and electricity ha AreaWoodyPerennials (Electricity) Total area of woody perennials for electricity ha and employed parallel programming techniques in the sensitivity outputs in order to justify our use of the variance decomposition and uncertainty analyses (Bryan, 2013; Zhao et al., 2014). Parameter method and GSA work as very small CV values indicate that un- set generation was undertaken using Matlab R2013b (The Math- derstanding the contribution of model inputs to model output Works Inc., USA). For each parameter set, an individual LUTO variance would be of little value (Cosenza et al., 2014). CV was also simulation was parameterised, and a compute job was created and used to describe the degree of dispersion and shape of output submitted to CSIRO's computing clusters, and executed in parallel. frequency distributions under different scenarios. Second, to test The sensitivity and uncertainty analyses were also conducted using for significant differences between the outputs from different Matlab R2013b (The MathWorks Inc., USA). scenarios, the non-parametric KruskaleWallis test (Corder and The eFAST sampling procedure implements a sinusoidal func- Foreman, 2009) was used due to the lack of normality in the tion of a particular frequency ui associated with the input param- data. The level of significance was a ¼ 0.05 in both tests. Last, we eter xi. The frequency assigned to the parameter is based on the visualised between-scenario differences in uncertainty using a sample from 1 to the number of samples per search curve, Ns. combined violin and box plot. Saltelli et al. (1999) suggested the minimum value for Ns is 65, and optimal values of ui are between 16·Nr and (16·Nr þ 48·Nr), where Nr 3.3. Correlation analyses of between-scenario parameter sensitivity is the number of resamples. We used 145 samples per search curve without resampling (i.e. Ns ¼ 145,Nr ¼ 1) where ui was calculated as Using eFAST in a GSA, we quantified the sensitivity of model 18. Thus, the total number of model simulations was 7250 for 50 outputs to uncertainty in model input parameters under the four parameters (i.e. 145  50) for each scenario. The required number of scenarios. To quantify the differences between sensitivity indices model simulations was confirmed by a convergence test, which under different scenarios, we used the non-parametric Spearman's incrementally increased the number of model runs and quantified rank correlation test which does not require normality in the data the output difference between two subsequent increments (Song (Jackson, 2012). The correlation coefficient describes the strength of et al., 2013; Cosenza et al., 2014). The simulation results were association between sensitivity index sets. Following Walker and used for both uncertainty and sensitivity analyses. Almond (2010), we defined the relationship as strong when an r We identified influential input parameters by calculating both value varies between ±(0.8~1), as moderate when r varies between the first-order sensitivity index Si and total sensitivity index STi. In ±(0.6~0.79), as weak when r varies between ±(0.4~0.59), and as line with Cosenza et al. (2014), for each output variable, a threshold little or no association when r varies between ±(0.0~0.39). of Si > 0.01 and STi > 0.1 was imposed to identify influential pa- rameters. We displayed the results of the eFAST sensitivity analyses in grid diagrams (Song et al., 2012). 3.4. Robust sensitivity indicators A robust sensitivity indicator (Scriterion RTi ) of each input parameter 3.2. Uncertainty analysis and between-scenario differences in (xi) for each model output was calculated based on the total model outputs sensitivity indices (STi,L1, STi,M3, STi,M2, and STi,H3) for the four sce- narios. We used four criteria from decision theory (maximax, Uncertainty analysis aims to depict the entire set of possible weighted average, minimax regret, and limited degree of confi- model outcomes, together with their associated frequencies of dence) to calculate robust sensitivity indicators. occurrence (Loucks et al., 2005). We conducted uncertainty ana- lyses for investigating the between-scenario differences of outcome (1) Maximax uncertainty derived from the variation in input parameters with eFAST. First, we calculated coefficients of variation (CV) of model The maximax criterion (Anderson et al., 2012) is an aggressive  L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 159 decision making rule under conditions of uncertainty. The decision lowest maximum regret value is selected. criterion finds the maximum possible payoff (profit or loss) for each
 alternative, and then selects the alternative with the greatest   min max ra;b /j (6) maximum payoff. In decision theory, payoffs are the outcomes for a b any combination of an alternative a (scenario a) and state of nature b (scenario b that the future will follow). Here, payoff pa, b is defined where / is a selection symbol that identifies the final alternative as the total index difference between two scenarios (a and b). scenario j according to the identification condition of Therefore, for an input parameter xi, payoff pa, b is defined as: minðmax ðra;b ÞÞ. a b Finally, the total sensitivity index STi, j under the scenario j was pa;b ¼ STi;a STi;b (1) selected to represent the robust sensitivity indicator Sreg for each RTi The payoffs associated with all possible combinations of alter- input parameter xi (i ¼ 1, 2,…,k) for unknown future conditions. natives and states of nature constitute a payoff table. The maximum reg SRTi ¼ STi;j (7) possible payoff for each alternative is selected first, and then the alternative j with the greatest maximum payoff is identified, such that: (4) Limited degree of confidence (LDC)
 The LDC criterion (Lempert and Collins, 2007; McInerney et al.,  max max pa;b /j (2) a b 2012) was used to balance the discounted sum of total sensitivity indices Savg RTi and the minimax regret performance. The criterion was where / is a selection symbol that identifies the final alternative built on the decision-maker only having a limited degree of confi- scenario j according to the identification condition of dence in the best estimate probability function of the occurrence of maxðmax ðpa;b ÞÞ. a the scenarios. The robust sensitivity indicator Sldc RTi can be repre- b Thus, the total index STi,j under scenario j is the robust sensitivity sented as: indicator Smax RTi for the input parameter xi (i ¼ 1, 2,…,k), as shown avg below: Sldc RTi ¼ g·SRTi þ ð1 gÞ·Sreg RTi (8) Smax (3) where g lies between 0 and 1, and represents the decision-maker's RTi ¼ STi;j degree-of-confidence in the underlying probability distributions. In our case, for a model output, the maximax criterion ensures When g ¼ 0, this criterion is equivalent to the minimax regret and that no parameter that is influential in any scenario is omitted. when g ¼ 1 it is equivalent to the weighted average criterion. We Here, the effect of applying this criterion to an input parameter is set g ¼ 0.5 such that the four scenarios have equal occurrence equivalent to selecting the maximum total sensitivity index over all probability. four scenarios. 4. Results (2) Weighted average Of the 24 output variables, the five most important ones (Agri- We used a weighted average criterion to calculate the robust culturalProduction, CarbonSequestration, WaterIntercepted, Bio- avg sensitivity indicator SRTi by aggregating the four total sensitivity diversityServices, and EnergyBiofuel) were selected to illustrate how indices (STi,L1, STi,M3, STi,M2, and STi,H3), in order to identify the input the sensitivity of these outputs to sampled parameters behaved avg parameters most influential over all four scenarios. SRTi is the sum differently under the four scenarios, and how new robust indicator of the product of the weight ba of each scenario a and the total sets were developed. The full set of results is presented in the sensitivity index STi, a under this scenario. Supplementary Material Figs. B1eB2, Table B1, and Figs C1eC9. Savg X  RTi ¼ ba $STi; a (4) 4.1. Between-scenario differences in model outputs a where a ¼ L1, M3, M2, and H3. As we have no information on Out of the CV values of the 24 outputs under the four scenarios, occurrence probability, we specified equal weights for all four only the outputs AreaOtherCrops and AreaDairy had relatively low scenarios. CV values: approximately 0.04 and 0.1, respectively, across the four scenarios (Supplementary Material Fig. B1). The CV values in the (3) Minimax regret other output variables exceeded 0.2 under the four outlooks. Thus, for most outputs, there were substantial differences in the CV The minimax regret criterion (Anderson et al., 2012) was used to values of different scenarios, thereby affirming the need for sensi- minimise the regret of accepting sensitivity indices from a scenario tivity analysis of model outputs. which does not eventuate. In this criterion, the regret (opportunity Between-scenario uncertainty in the 24 outputs was significant loss) is used rather than payoff. Here, for each state of nature b under the specified variation in input parameters (Supplementary (scenario b), the regret ra, b is defined as the absolute difference Material Fig. B2). Some outputs such as ProductionBeef between the total index STi, a in an alternative scenario a and STi, b. (Supplementary Material Fig. B2 (d)) and AgriculturalProduction (Fig. 2) had a more normal distribution, while the distributions of  ra;b ¼ STi;a  STi;b  (5) others such as EnergyBiofuel (Supplementary Material Fig. B2(j) or Fig. 2) and AreaGrains (see Supplementary Material Fig. B2(1)) were The regret values from all possible combinations of alternative more skewed. Although these distributions appeared similar under scenarios and states of nature can form a regret table, where the the four scenarios, they differed in magnitude. For example, the maximum regret value for each alternative scenario can be iden- most common values for ProductionBeef were approximately tified as max ðra;b Þ and then the alternative scenario j with the calculated as 12.32, 15.49, 17.72, and 20.03 million head for L1, M3, b 160 L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 Fig. 2. Empirical frequency distributions for the five main outputs under four scenarios. The line red in the box is the median; the edges of the box are the lower hinge (the 25th percentile, Q1) and the upper hinge (the 75th percentile, Q3), and the whiskers extend to 1.5  (Q3 Q1). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.) M2, and H3, respectively. Significant between-scenario differences were found under the four scenarios. There were 33, 39, 46, and 43 for 23 of the model 24 outputs (all except Area- influential parameters under L1, M3, M2, and H3, respectively, for WoodyPerennials(Electricity)) were identified by the KruskaleWallis all outputs. The differences of the main and total sensitivity effects test (Supplementary Material Table B1). Therefore, ignoring sce- of the ten (or less) most influential input parameters for the 24 nario impacts on output uncertainty in a typical GSA could result in outputs are visualised in Supplementary Material Fig. C9(aex). a biased and unrepresentative view of model structure, perfor- Substantial differences were found between scenarios in the mance, parameter sensitivity, and output uncertainty. total sensitivity effects (STi) reflecting the influence of input pa- rameters on the five main outputs (Fig. 3 and Fig. 4). For the output 4.2. Sensitivity of outputs to input parameters under different AgriculturalProduction, the second most influential parameter var- scenarios ied between scenarios: CommodityPricePrimary for scenario L1, WheatGrainYield for scenarios M3 and H3, and AgriculturalYield for Supplementary Material Fig. C1eC8 reported the full results of H3. AgriculturalProductivity, WheatGrainYield, AgriculturalYield, and first-order and total sensitivity effects for all 50 input parameters CommodityPricePrimary were the top four most influential param- on the 24 output variables, under the four scenarios. Different eters for the four scenarios, except WheatGrainYield was non- numbers of influential parameters (i.e. where Si > 0.01 and ST i> 0.1) influential under L1.  L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 161 Fig. 3. Total sensitivity effects of 50 input parameters on the five outputs under the four scenarios. Colours in the grid cells represent the total sensitivity effects and numbers represent their rankings of influence. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.) Fig. 4. Si and STi for the influential input parameters for the five outputs. The parameters are ranked based on the maximum STi across the four scenarios. 162 L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 For CarbonSequestration, ten input parameters were influential, likely to be non-influential under other decision criteria, especially but the top three under the four scenarios included nine different under the minimax regret criterion whose purpose is to reduce the parameters (CarbSeqRateCP, AdoptionHurdleRate, and CarbSeqRisk- worst-case performance (i.e. selecting the wrong index that is not Buffer in L1; CarbonPricePath, LivestockPricePath, and Agricultur- robust to the four scenarios). For instance, the sensitivity indices for alProductivity in M3; CarbSeqRateCP, CarbonPricePath, and CarbSeqRateEP to the output CarbonSequestration, and Bio- TreeGrowthProductivity in M2; and CarbSeqRateEP, BiodiversityFund, diversityFund to the outputs CarbonSequestration, WaterIntercepted, and LivestockPricePath in H3). as well as BiodiversityServices were significant under H3 but insig- Nine parameters were influential for WaterIntercepted. Under L1, nificant under the other scenarios (Fig. 5). These parameters were AdoptionHurdleRate, CarbonPricePath, WaterInterceptionTrees were influential when the decision maker wants to identify all parame- influential. CarbonPricePath was the most important input under ters influential under any scenario (by applying the maximax cri- both M3 and M2. For M3, the influential parameters still included terion). However, the regrets (opportunity loss) were large when AdoptionHurdleRate, LivestockPricePath, and TreeGrowthProductivity. the future does not follow the trajectories of H3. Hence, a risk- For M2, the second most important parameter was CarbSeqRateCP, averse decision maker may regard these parameters as non- followed by AgriculturalProductivity, TreeGrowthProductivity, and influential (i.e. by applying the minimax regret criterion). AdoptionHurdleRate. BiodiversityFund was influential only in sce- The weighted average and the minimax regret criterion gener- nario H3. ally identified similar influential parameters, but differed in the Similarly, input parameters influencing BiodiversityServices were ranking of these parameters. This is because the two criteria differ heavily dependent upon the scenario. BiodiversityLevy, Carbon- in the aggregation method of total sensitivity indices over all sce- PricePath, and AdoptionHurdleRate significantly affected the output narios. The former focuses on identifying influential parameters under both scenarios L1 and M3. However, the ranking of these over all scenarios while the latter aims for minimising the regret of three parameters and their magnitudes of total effects were quite selecting the wrong alternative. For example, for the output Bio- different. More parameters were identified as influential under M2. diversityServices, CarbonPricePath had higher total sensitivity Under H3, the effect of BiodiversityLevy was negligible, but Adop- indices over the four scenarios than BiodiversityLevy. Under M3 and tionHurdleRate, BiodiversityFund, LivestockPricePath, and Oil- M2, CarbonPricePath's STi was >40% and BiodiversityLevy's STi was PricePath were important. z20% (Fig. 5). Therefore, CarbonPricePath and BiodiversityLevy For EnergyBiofuel, seven influential parameters were identified ranked the first and second, respectively, when the weighted under L1, with WheatPrice, CommodityPricePrimary, and Agri- average criterion was applied, but this ranking was reversed under culturalProductivity the top three. Under M3, five parameters were the minimax regret criterion due to larger opportunity loss in case influential (WheatPrice, AgriculturalProductivity, and CropPricePath of selecting a wrong alternative for parameter CarbonPricePath. In held the top three positions). Seven parameters were significant our case, the LDC criterion balanced the effects of the weighted under M2 (OilPricePath, AgriculturalProductivity, and WheatPrice average and the minimax regret criteria. ranked top three). Six parameters were influential under H3, with WheatPrice, OilPricePath, and AgriculturalProductivity the top three. 5. Discussion Correlations in the sensitivity of outputs to model inputs be- tween scenarios were varied. For CarbonSequestration, Water- 5.1. Influence of deep uncertainty on output uncertainty and Intercepted, and BiodiversityServices, there was a weak relationship parameter sensitivity between the sensitivity index pairs involving H3 (Table 4). 60% of all scenario pairings for the five main model outputs showed moder- The future state of the world (such as future global economy, ate to weak correlation in the influence of input parameters. population, greenhouse gas emissions, and climate) is character- ized by deep uncertainty for which “there is no objective measure 4.3. Robust sensitivity analyses of the probability” (Knight, 1921; Ben-Haim, 2001). We performed global sensitivity and uncertainty analyses of the LUTO modelda Given the difference in input parameter influence and model large, integrated social-ecological system model of Australian land sensitivities demonstrated above, we present four sets of robust use futures, under deep uncertainty. The type of uncertainty sur- sensitivity indicators by applying four decision criteria. For Agri- rounding future drivers is probabilistically intractable and we culturalProduction, the four decision criteria resulted in the same considered it using internally consistent scenarios, each of which is parameter ranking in the sensitivity indicators: Agricultur- a structured account of a plausible climate, population, carbon, alProductivity, WheatGrainYield, AgriculturalYield, and Commodity- energy, and economic future. We used eFAST which has proven to PricePrimary (Fig. 5). be amongst the most reliable and efficient global sensitivity anal- Variation was also apparent for the other outputs. In particular, ysis methods. Varying the scenario parameters as in typical GSA parameters whose sensitivity index was only high in one scenario, (Sobol', 1993; Saltelli et al., 1999) was not appropriate as it violated could be identified as influential by the maximax criterion, but is internal consistency and risked running the model under implau- sible or impossible conditions. GSA undertaken for each scenario found significant differences in the uncertainty of outputs, and in Table 4 the influence of model inputs. Correlation coefficients of between-scenario sensitivity indices for the five main We found that model outputs resulting from variation in model outputs. inputs in GSA were significantly different under the four scenarios Model output Scenario pair as indicated by the CV test and KruskaleWallis test. For some out- L1~M3 L1~M2 L1~H3 M3~M2 M3~H3 M2~H3 puts, different scenarios even led to different distributional shapes. Scenarios captured key ingredients of deep uncertainty about future AgriculturalProduction 0.89 0.88 0.81 0.94 0.95 0.94 CarbonSequestration 0.78 0.67 0.47 0.78 0.43 0.43 drivers and the LUTO model responded in a strongly non-linear way WaterIntercepted 0.81 0.75 0.42 0.90 0.51 0.42 to these drivers. This could be seen in some outputs where the BiodiversityServices 0.87 0.84 0.46 0.92 0.49 0.53 impact of a parameter's interactions with other parameters on the EnergyBiofuel 0.89 0.63 0.66 0.66 0.72 0.77 outputs (STi Si) was much greater than that of the parameter's own Note: the weak correlation relationships are marked in bold. contribution (Si). Different scenarios provided different drivers of  L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 163 Fig. 5. Robust sensitivity indicators for the top ten most influential input parameters for the five key output variables using the four decision criteria. change to the complex non-linear system and led to variation in management strategies. We believe that none of these such exer- uncertainty in model outputs generated from variation in param- cises explored the scope and the distribution of model outcomes eter inputs. This is a novel and important contribution. Most sce- under different scenarios resulting from variation in model inputs. nario exercises (e.g., Millennium Ecosystem Assessment, 2005a; The finding that significant differences exist in parameters Moss et al., 2010; van Vuuren et al., 2011; Bateman et al., 2013; identified as influential and non-influential, in parameter impor- Howells et al., 2013), have been conducted to help decision tance rankings, and in the magnitude of first-order and total effects makers understand a range of possible futures and create robust under deep uncertainty, has not been reported previously. While 164 L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 several global sensitivity analyses have assessed model uncertainty efficient under the kind of deep uncertainty characteristic of social- and parameter sensitivity under future conditions (e.g., Anderson ecological systems. et al., 2014; Butler et al., 2014), none were cognisant of the influ- Decisions must be made about how to cope with and take ence of deep uncertainty on the results. These findings are impor- advantage of the significant differences that exist in uncertainty of tant because they provide solid evidence supporting the need for outputs and sensitivity of inputs between scenarios. New, robust robust GSA to guide further data collection and analyses when technical methods have recently emerged to contribute to the faced with deep uncertainty. Exploring the dependence of GSA robust risk analysis and decision making in the presence of deep results on model scenarios (deeply uncertain conditions under uncertainty, including model ensembles, adaptive management, which models are run) is critical. Failure to do so could result in the robust optimisation, and decision algorithms (Cox, 2012). These targeting of parameters for modelling, data collection, and refine- methods usually employ multiple models and scenarios, and with ment that may not be influential under certain scenarios. Critically, these methods, the decision maker needs to act intelligently on the it may also lead to the exclusion from these efforts of parameters basis of possible future changes. When the degree of uncertainty is that have been found to be non-influential, but which are highly high and uncertainty is not controllable (for instance, unknown influential under alternative unanalysed scenarios. It is likely that futures), scenario planning and decision algorithms are effective these findings are generalisable beyond the LUTO model and can be ways of coping (Peterson et al., 2003; Cox, 2012). Application of extrapolated to other social-ecological models that run under deep these and other robust technical methods and integrating them uncertainty. into a formal set of mathematical and computational processes for Here, we focussed on the most influential parameters that need robust global sensitivity analysis seems a productive path for future to be prioritised for future investment in modelling-related activ- research. ities. A common alternate use of the same information is in the identification of non-influential parameters for the purpose of 5.3. Limitations and future directions model simplification (Saltelli et al., 2006). For the LUTO model, few non-influential parameters were identified under each scenario There are two main limitations in our GSA experiments. First, to and very different results were obtained between scenarios. The reduce the parameter dimensionality and computational demands, result was due to the inherent complexity of the LUTO model, the variation of temporal and spatial parameters were simplified by external influences (expressed by scenarios) on the model, and the introducing a multiplier ranging from 0.7 to 1.3 (i.e. ±30%) uni- thresholds for identifying non-influential parameters. This result formly applied to all time-series and spatial parameters. This confirms the importance of considering deep uncertainty when the ignored the variation characteristics and relations of parameters focus of GSA is on identifying non-influential model parameters that may change non-uniformly over time and/or space. Some (i.e. parameter screening). recent progress has been made in investigating the effect of spatial parameters on sensitivity (e.g., Dong et al., 2015) but more effort is 5.2. Decision making for deeply uncertain futures required. In addition, these variation ranges could have a significant influence on the parameter sensitivity (Wang et al., 2013) and more We needed to quantify sensitivity in a way that was robust to effort to refine parameter boundary conditions of influential pa- deep uncertainty and can be used to reliably inform investment in rameters would further improve the GSA results. data collection, modelling, and refining of parameter estimates. We The second major limitation is the likelihood that the definition proposed four decision criteria to find robust sensitivity indices of the four global outlooks as scenarios was not sufficient to capture that work acceptably well under all scenarios. Following decision the range of future uncertainty in all parameters (Bryant and theory, we calculated four new sensitivity indicators based on the Lempert, 2010). Recently, a crucial contribution to decision mak- decision objective, which incorporated the decision maker's atti- ing under deep uncertainty are bottom-up approaches of scenario tude to risk. No decision criterion is the right or best choice. Success discovery (Herman et al., 2015) which advocate the generation and in the future is based on a decision's future success, which we analysis of many (e.g., thousands of) scenarios. Examples include cannot know in advance. Our aim was not to make a judgement Robust Decision Making (Groves and Lempert, 2007; Lempert and about which criterion should be applied by the decision maker, but Collins, 2007; Bryant and Lempert, 2010; Kwakkel et al., 2014), rather to provide a number of options to support decision-making Decision Scaling (Brown et al., 2012; Brown and Wilby, 2012), Info- under deep uncertainty. A specific criterion can be selected based Gap (Ben-Haim, 2004; Matrosov et al., 2013), and Many-Objective on the objective and level of risk aversion of the decision maker. Robust Decision Making (Kasprzyk et al., 2013; Herman et al., The maximax criterion is ‘aggressive’ and suitable for the decision 2014). One challenge with using scenario discovery techniques to maker who is not willing to omit any possible influential parameter. incorporate many scenarios into sensitivity analysis of complex The minimax regret criterion is relatively conservative and fits systems models is the sheer amount of effort involved in their those who want to minimise the regret of the worst case- specification. In our case, scenario parameters were quantified by daccepting sensitivity indices from a state of world that does not integrated assessment to produce the input parameters shown in eventuate. The weighted average criterion is recommended to Table 1 which is in itself a highly technical and effortful task. those who expect to convert ‘deep uncertainty’ to ‘probabilistic Another challenge with incorporating many scenarios is that it adds uncertainty’ and have their own confidence in the assessment for significant computational load to global sensitivity analyses that the probabilities of sensitivity indices from different scenarios. The are already pushing present-day computational limits (Song et al., LDC criterion is balance of weighted average and minimax regret, 2012; Zhao et al., 2014). For example, for computational tracta- and the weighting in LDC can be explained as a measure of the bility in this study we used a coarse resolution version of the LUTO decision maker's confidence in the occurrence probabilities of model (9.9 km2 grid cells instead of 1.2 km2 at the highest model different future scenarios. Choice of decision criterion can make the resolution). Each model run took approximately 1e2 h, totalling at decision maker comfortable with the ambiguity of an open future least 7250 h (145  50  1) to undertake the eFAST analysis under (Gao and Hailu, 2012, 2013). Robust sensitivity metrics allowed us one scenario. Analysing hundreds or thousands of scenarios pre- to identify those parameters that are most influential for specific sents a significant computational challenge, even at coarse reso- outputs irrespective of scenario. This ensures that focussing lution and with access to high performance computing (Bryan, attention on the most robustly influential parameters will be most 2013). To overcome this barrier, we are currently investigating  L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 165 the potential of screening methods as robust GSA approaches with Binner, A., Crowe, A., Day, B.H., Dugdale, S., 2013. Bringing ecosystem services into economic decision-making: land use in the United Kingdom. Science 341 greatly reduced computational demand. We incorporated scenarios (6141), 45e50. into the p-level sampling space (p is the number of levels to which Ben-Haim, Y., 2001. Info-Gap Decision Theory. Academic Press, San Diego, CA. each dimension of the parameter space is divided) of Morris' Ben-Haim, Y., 2004. Uncertainty, probability and information-gaps. Reliab. Eng. elementary effects method (Gao and Bryan, in press). However, as Syst. Saf. 85 (1e3), 249e266. Bohensky, E.L., Reyers, B., Van Jaarsveld, A.S., 2006. Future ecosystem services in a the choice of p is strictly linked to the choice of r (the number of southern African river basin: a scenario planning approach to uncertainty. trajectories sampled), the increase of scenarios inevitably adds Conserv. Biol. 20 (4), 1051e1061. computational cost. Further advances are required to overcome Brown, C., Ghile, Y., Laverty, M., Li, K., 2012. Decision scaling: linking bottom-up vulnerability analysis with climate projections in the water sector. Water these technical and computational challenges and enable scenarios Resour. Res. 48 (9), W09537. to capture a greater range of future uncertainty in sensitivity ana- Brown, C., Wilby, R.L., 2012. An alternate approach to assessing climate risks. Eos lyses of complex models via scenario discovery techniques, and Trans. Am. Geophys. Union 93 (41), 401e402. Bryan, B.A., 2013. High-performance computing tools for the integrated assessment thereby improve the robustness of produced sensitivity metrics. and modelling of socialeecological systems. Environ. Model. Softw. 39, 295e303. 6. Conclusion Bryan, B.A., Crossman, N.D., King, D., Meyer, W.S., 2011. Landscape futures analysis: assessing the impacts of environmental targets under alternative spatial policy options and future scenarios. Environ. Model. Softw. 26 (1), 83e91. Using the eFAST method, we produced comprehensive model Bryan, B.A., Nolan, M., Harwood, T.D., Connor, J.D., Navarro-Garcia, J., King, D., diagnostics of a large, integrated social-ecological systems model Summers, D.M., Newth, D., Cai, Y., Grigg, N., Harman, I., Crossman, N.D., Grundy, M.J., Finnigan, J.J., Ferrier, S., Williams, K.J., Wilson, K.A., Law, E.A., under conditions of deep uncertainty characterised by four global Hatfield-Dodds, S., 2014. Supply of carbon sequestration and biodiversity ser- change scenarios, each of which represents a set of plausible, vices from Australia's agricultural land under global change. Glob. Environ. internally consistent future conditions. We found significant dif- Change 28, 166e181. Bryan, B.A., Nolan, M., McKellar, L., Connor, J.D., Newth, D., Harwood, T., King, D., ferences between scenarios in model outputs that resulted from Navarro, J., Cai, Y., Gao, L., Grundy, M., Graham, P., Ernst, A., Dunstall, S., Stock, F., variation in model inputs. The contributions of input parameters to Brinsmead, T., Harman, I., Grigg, N.J., Battaglia, M., Keating, B., Wonhas, A., the scope and distribution of each model output were also highly Hatfield-Dodds, S., 2015. Land-use and sustainability under intersecting global change and domestic policy scenarios: trajectories for Australia to 2050. Global scenario-dependent. The GSA results revealed that global scenarios Environmental Change (in review). had a significant effect on the influential and non-influential pa- Bryant, B.P., Lempert, R.J., 2010. Thinking inside the box: a participatory, computer- rameters, parameter importance ranking, and the magnitude of assisted approach to scenario discovery. Technol. Forecast. Soc. Change 77 (1), 34e49. both first-order and total effects. Four criteria from decision theory Butler, M.P., Reed, P.M., Fisher-Vanden, K., Keller, K., Wagener, T., 2014. Identifying were used to successfully integrate parameter sensitivity estimates parametric controls and dependencies in integrated assessment models using across scenarios and provide options for decision makers to global sensitivity analysis. Environ. Model. Softw. 59, 10e29. determine new sensitivity indicators that were robust to deeply Chen, L., Tian, Y., Cao, C., Zhang, S., Zhang, S., 2012. Sensitivity and uncertainty analyses of an extended ASM3-SMP model describing membrane bioreactor uncertain conditions. The decision-maker can then evaluate the operation. J. Membr. Sci. 389, 99e109. decision criteria against their own risk preferences and more Confalonieri, R., Bellocchi, G., Tarantola, S., Acutis, M., Donatelli, M., Genovese, G., robustly quantify sensitivity for reliably informing further invest- 2010. Sensitivity analysis of the rice model WARM in Europe: exploring the effects of different locations, climates and methods of analysis on model ment in data collection, modelling, and refining of parameter es- sensitivity to crop parameters. Environ. Model. Softw. 25 (4), 479e488. timates. These findings are of general value and can be extrapolated Connor, J.D., Bryan, B.A., Nolan, M., Stock, F., Gao, L., Dunstall, S., Graham, P., to other models that must run under deep uncertainty. The pro- Ernst, A., Newth, D., Grundy, M., Hatfield-Dodds, S., 2015. Modelling Australian land use competition and ecosystem services with food price feedbacks at high posed methods are a valuable addition to global sensitivity analyses spatial resolution. Environ. Model. Softw. 69, 141e154. that can also increase our understanding of the effects of deep Corder, G.W., Foreman, D.I., 2009. Nonparametric Statistics for Non-statisticians: a uncertainty on output uncertainty and parameter sensitivity, Step-by-step Approach. Wiley, New Jersey, USA. Cosenza, A., Mannina, G., Vanrolleghem, P.A., Neumann, M.B., 2013. Global sensi- incorporate the decision maker's risk preference into modelling- tivity analysis in wastewater applications: a comprehensive comparison of related activities, and obtain greater resilience of decisions to different methods. Environ. Model. Softw. 49, 40e52. surprise. Cosenza, A., Mannina, G., Vanrolleghem, P.A., Neumann, M.B., 2014. Variance-based sensitivity analysis for wastewater treatment plant modelling. Sci. Total Envi- ron. 470e471, 1068e1077. Acknowledgements Cox, L.A., 2012. Confronting deep uncertainties in risk analysis. Risk Anal. 32 (10), 1607e1629. Davies-Barnard, T., Valdes, P.J., Singarayer, J.S., Pacifico, F.M., Jones, C.D., 2014. Full We are grateful for the support of CSIRO Agriculture, CSIRO Land effects of land use change in the representative concentration pathways. En- and Water, and the Australian National Outlook initiative. We also viron. Res. Lett. 9 (11), 114014. appreciate the comments of three anonymous reviewers which DeJonge, K.C., Ascough Ii, J.C., Ahmadi, M., Andales, A.A., Arabi, M., 2012. Global sensitivity and uncertainty analysis of a dynamic agroecosystem model under have greatly improved this manuscript. different irrigation treatments. Ecol. Model. 231, 113e125. Dong, M., Bryan, B.A., Connor, J.D., Nolan, M., Gao, L., 2015. Land use mapping error Appendix A. Supplementary data introduces strongly-localised, scale-dependent uncertainty into land use and ecosystem services modelling. Ecosyst. Serv. 15, 63e74. Freni, G., Mannina, G., 2010. Uncertainty in water quality modelling: the applica- Supplementary data related to this article can be found at http:// bility of variance decomposition approach. J. Hydrol. 394 (3e4), 324e333. dx.doi.org/10.1016/j.envsoft.2015.11.001. Gan, Y., Duan, Q., Gong, W., Tong, C., Sun, Y., Chu, W., Ye, A., Miao, C., Di, Z., 2014. A comprehensive evaluation of various sensitivity analysis methods: a case study with a hydrological model. Environ. Model. Softw. 51, 269e285. References Gao, L., Bryan, B.A., 2015. Incorporating deep uncertainty into the elementary ef- fects method for robust global sensitivity analysis. Ecol. Model. http:// Anderson, B., Borgonovo, E., Galeotti, M., Roson, R., 2014. Uncertainty in climate dx.doi.org/10.1016/j.ecolmodel.2015.10.016 (in press). change modeling: can global sensitivity analysis be of help? Risk Anal. 34 (2), Gao, L., Connor, J.D, Dillon, P., 2014. The economics of groundwater replenishment 271e293. for reliable urban water supply. Water 6 (6), 1662e1670. Anderson, D.R., Sweeney, D.J., Williams, T.A., Camm, J.D., Martin, R.K., 2012. An Gao, L., Hailu, A., 2012. Ranking management strategies with complex outcomes: an Introduction to Management Science: quantitative Approaches to Decision AHP-fuzzy evaluation of recreational fishing using an integrated agent-based Making. Cengage Learning, Mason, OH, USA. model of a coral reef ecosystem. Environ. Model. Softw. 31, 3e18. Annoni, P., Brüggemann, R., Saltelli, A., 2011. Partial order investigation of multiple Gao, L., Hailu, A., 2013. Identifying preferred management options: an integrated indicator systems using variance-based sensitivity analysis. Environ. Model. agent-based recreational fishing simulation model with an AHP-TOPSIS evalu- Softw. 26 (7), 950e958. ation method. Ecol. Model. 249, 75e83. Bateman, I.J., Harwood, A.R., Mace, G.M., Watson, R.T., Abson, D.J., Andrews, B., Groves, D.G., Lempert, R.J., 2007. A new analytic method for finding policy-relevant 166 L. Gao et al. / Environmental Modelling & Software 76 (2016) 154e166 scenarios. Glob. Environ. Change 17 (1), 73e85. Uncertainty in Quantitative Risk and Policy Analysis. Cambridge University Hatfield-Dodds, S., Brinsmead, T., Bryan, B., Graham, P., Grundy, M., Harwood, T., Press, Cambridge, UK. Newth, D., Schandl, H., Wonhas, A., Adams, P., McKellar, L.B., Cai, Y., Ferrier, S., Moss, R.H., Edmonds, J.A., Hibbard, K.A., Manning, M.R., Rose, S.K., van Vuuren, D.P., Finnigan, J., Hanslow, K., McCallum, R., Nolan, M., Prosser, I., Smith, M.S., Carter, T.R., Emori, S., Kainuma, M., Kram, T., Meehl, G.A., Mitchell, J.F.B., Baynes, T., Chiew, F., Connor, J., Geschke, A., Grigg, N., Harman, I., Hayward, J., Nakicenovic, N., Riahi, K., Smith, S.J., Stouffer, R.J., Thomson, A.M., Weyant, J.P., Keating, B., King, D., Lenzen, M., Lonsdale, M., McCrae, R., Garcia, J.N., Owen, A., Wilbanks, T.J., 2010. The next generation of scenarios for climate change Raison, J., Reedman, L., Smith, M.H., Summers, D., Whetton, P., 2015. CSIRO research and assessment. Nature 463 (7282), 747e756. Australian National Outlook 2014: Living Standards, Resource Use, Environ- Nossent, J., Elsen, P., Bauwens, W., 2011. Sobol' sensitivity analysis of a complex mental Performance and Economic Activity, 1970-2050. CSIRO, Canberra. environmental model. Environ. Model. Softw. 26 (12), 1515e1525. Herman, J.D., Reed, P.M., Zeff, H.B., Characklis, G.W., 2015. How should robustness be Peterson, G.D., Cumming, G.S., Carpenter, S.R., 2003. Scenario planning: a tool for defined for water systems planning under change? J. Water Resour. Plan. conservation in an uncertain world. Conserv. Biol. 17 (2), 358e366. Manag. 141 (10), 04015012. Polasky, S., Carpenter, S.R., Folke, C., Keeler, B., 2011. Decision-making under great Herman, J.D., Zeff, H.B., Reed, P.M., Characklis, G.W., 2014. Beyond optimality: uncertainty: environmental management in an era of global change. Trends multistakeholder robustness tradeoffs for regional water portfolio planning Ecol. Evol. 26 (8), 398e404. under deep uncertainty. Water Resour. Res. 50 (10), 7692e7713. Popper, S.W., Lempert, R.J., Bankes, S.C., 2005. Shaping the future. Sci. Am. 292, Howells, M., Hermann, S., Welsch, M., Bazilian, M., Segerstrom, R., Alfstad, T., 66e71. Gielen, D., Rogner, H., Fischer, G., van Velthuizen, H., Wiberg, D., Young, C., Quade, E.S., 1989. Analysis for Public Decisions, third ed. Elsevier Science, New York. Roehrl, R.A., Mueller, A., Steduto, P., Ramma, I., 2013. Integrated analysis of Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M., Tarantola, S., 2010. climate change, land-use, energy and water strategies. Nat. Clim. Change 3 (7), Variance based sensitivity analysis of model output. Design and estimator for 621e626. the total sensitivity index. Comput. Phys. Commun. 181 (2), 259e270. IPCC, 2000. Special Report on Emissions Scenarios. A Special Report of Working Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Group III of Tiie Intergovernmental Panel on Climate Change. Cambridge Uni- Tarantola, S., 2008. Global Sensitivity Analysis: the Primer. John Wiley & Sons, versity Press, Cambridge, UK. West Susses, England. IPCC, 2007. In: Writing Team, Core, Pachauri, R.K., Reisinger, A. (Eds.), Climate Saltelli, A., Ratto, M., Tarantola, S., Campolongo, F., 2006. Sensitivity analysis prac- Change 2007: Synthesis Report. Contribution of Working Groups I, II and III to tices: strategies for model-based inference. Reliab. Eng. Syst. Saf. 91 (10e11), the Fourth Assessment Report of the Intergovernmental Panel on Climate 1109e1125. Change. IPCC, Geneva, Switzerland, p. 104. Saltelli, A., Tarantola, S., Chan, K.P.S., 1999. A quantitative model-Independent Jackson, S.L., 2012. Research Methods and Statistics: a Critical Thinking Approach. method for global sensitivity analysis of model output. Technometrics 41 (1), Cengage Learning, Belmont, CA, USA. 39e56. Kasprzyk, J.R., Nataraj, S., Reed, P.M., Lempert, R.J., 2013. Many objective robust Schaldach, R., Alcamo, J., Koch, J., Ko €lking, C., Lapola, D.M., Schüngel, J., Priess, J.A., decision making for complex environmental systems undergoing change. En- 2011. An integrated approach to modelling land-use change on continental and viron. Model. Softw. 42, 55e71. global scales. Environ. Model. Softw. 26 (8), 1041e1051. Kirby, J.M., Connor, J., Ahmada, M.D., Gao, L., Mainuddin, M., 2014. Climate change Shin, M.-J., Guillaume, J.H.A., Croke, B.F.W., Jakeman, A.J., 2013. Addressing ten and environmental water reallocation in the Murray-Darling Basin: impacts on questions about conceptual rainfallerunoff models with global sensitivity an- flows, diversions and economic returns to irrigation. J. Hydrol. 518, 120e129. alyses in R. J. Hydrol. 503, 135e152. Kirby, M., Connor, J., Ahmad, M.-u.D., Gao, L., Mainuddin, M., 2015. Irrigator and Sobol', I.M., 1993. Sensitivity analysis for nonlinear mathematical models. Math. environmental water management adaptation to climate change and water Model. Comput. Exp. 1, 407e414. reallocation in the MurrayeDarling Basin. Water Econ. Policy 01 (03), 1550009. Song, X., Bryan, B.A., Almeida, A.C., Paul, K.I., Zhao, G., Ren, Y., 2013. Time-dependent Knight, F.H., 1921. Risk, Uncertainty, and Profit. Houghton Mifflin, Boston, MA. sensitivity of a process-based ecological model. Ecol. Model. 265, 114e123. Kwakkel, J., Haasnoot, M., Walker, W., 2014. Developing dynamic adaptive policy Song, X., Bryan, B.A., Paul, K.I., Zhao, G., 2012. Variance-based sensitivity analysis of pathways: a computer-assisted approach for developing adaptive strategies for a forest growth model. Ecol. Model. 247, 135e143. a deeply uncertain world. Clim. Change 1e14. Ullman, D.G., 2006. Making Robust Decisions: Decision Management for Technical, Lempert, R.J., Collins, M.T., 2007. Managing the risk of uncertain threshold re- Business, & Service Teams. Trafford Publishing, Victoria, BC, Canada. sponses: comparison of robust, optimum, and precautionary approaches. Risk Van Oijen, M., Rougier, J., Smith, R., 2005. Bayesian calibration of process-based Anal. 27 (4), 1009e1026. forest models: bridging the gap between models and data. Tree Physiol. 25 Loucks, D.P., van Beek, E., Stedinger, J.R., Dijkman, J.P.M., Villars, M.T., 2005. Water (7), 915e927. Resources Systems Planning and Management: an Introduction to Methods, van Rossum and The Python Community, 2013. The Python Programming Lan- Models and Applications. UNESCO, Paris, France. guage: Version 2.7.5. The Python Software Foundation. Mahmoud, M., Liu, Y., Hartmann, H., Stewart, S., Wagener, T., Semmens, D., van Vuuren, D., Edmonds, J., Kainuma, M., Riahi, K., Thomson, A., Hibbard, K., Stewart, R., Gupta, H., Dominguez, D., Dominguez, F., Hulse, D., Letcher, R., Hurtt, G., Kram, T., Krey, V., Lamarque, J.-F., Masui, T., Meinshausen, M., Rashleigh, B., Smith, C., Street, R., Ticehurst, J., Twery, M., van Delden, H., Nakicenovic, N., Smith, S., Rose, S., 2011. The representative concentration Waldick, R., White, D., Winter, L., 2009. A formal framework for scenario pathways: an overview. Clim. Change 109 (1e2), 5e31. development in support of environmental decision-making. Environ. Model. Vanuytrecht, E., Raes, D., Willems, P., 2014. Global sensitivity analysis of yield Softw. 24 (7), 798e808. output from the water productivity model. Environ. Model. Softw. 51, Makler-Pick, V., Gal, G., Gorfine, M., Hipsey, M.R., Carmel, Y., 2011. Sensitivity 323e332. analysis for complex ecological models e a new approach. Environ. Model. Varella, H., Gue rif, M., Buis, S., 2010. Global sensitivity analysis measures the quality Softw. 26 (2), 124e134. of parameter estimation: the case of soil parameters and a crop model. Environ. Marino, S., Hogue, I.B., Ray, C.J., Kirschner, D.E., 2008. A methodology for performing Model. Softw. 25 (3), 310e319. global uncertainty and sensitivity analysis in systems biology. J. Theor. Biol. 254 Vezzaro, L., Mikkelsen, P.S., 2012. Application of global sensitivity analysis and (1), 178e196. uncertainty quantification in dynamic modelling of micropollutants in storm- Matrosov, E.S., Woods, A.M., Harou, J.J., 2013. Robust decision making and Info-Gap water runoff. Environ. Model. Softw. 27e28, 40e51. decision theory for water resource system planning. J. Hydrol. 494 (0), 43e58. Walker, J., Almond, P., 2010. Interpreting Statistical Findings. McGraw-Hill Educa- McInerney, D., Lempert, R., Keller, K., 2012. What are robust strategies in the face of tion, Berkshire, England. uncertain climate threshold responses? Clim. Change 112 (3e4), 547e568. Walker, W.E., Marchau, V.A.W.J., Swanson, D., 2010. Addressing deep uncertainty Miao, Z., Lathrop Jr., R.G., Xu, M., La Puma, I.P., Clark, K.L., Hom, J., Skowronski, N., using adaptive policies: Introduction to section 2. Technol. Forecast. Soc. Van Tuyl, S., 2011. Simulation and sensitivity analysis of carbon storage and Change 77 (6), 917e923. fluxes in the New Jersey Pinelands. Environ. Model. Softw. 26 (9), 1112e1122. Wang, J., Li, X., Lu, L., Fang, F., 2013. Parameter sensitivity analysis of crop growth Millennium Ecosystem Assessment, 2005a. Ecosystems and Human Well-being: models based on the extended Fourier Amplitude Sensitivity Test method. Scenarios. Island Press, Washington, DC. Environ. Model. Softw. 48, 171e182. Millennium Ecosystem Assessment, 2005b. Ecosystems and Human Well-being: Wilkinson, A., Kupers, R., 2013. Living in the futures. Harv. Bus. Rev. 91 (5), 119e127. Scenarios, vol. 2. Island Press, Washington, DC. Wintle, B.A., Runge, M.C., Bekessy, S.A., 2010. Allocating monitoring effort in the Morgan, M.G., Henrion, M., 1992. Uncertainty: a Guide to Dealing with Uncertainty face of unknown unknowns. Ecol. Lett. 13 (11), 1325e1337. in Quantitative Risk and Policy Analysis. Cambridge University Press, Cam- Zhao, G., Bryan, B.A., Song, X., 2014. Sensitivity and uncertainty analysis of the bridge, UK. APSIM-wheat model: interactions between cultivar, environmental, and man- Morgan, M.G., Henrion, M., Small, M., 1992. Uncertainty: a Guide to Dealing with agement parameters. Ecol. Model. 279, 1e11.