Abstract
Detailed understanding of the potential local or regional implications of climate change is required to guide decision- and policy-makers when developing adaptation strategies and designing infrastructure solutions suitable for potential future conditions. Impact models that translate potential future climate conditions into variables of interest (such as drought or flood risk) are needed to create the required causal connection between climate and impact for scenario-based analyses. Recent studies suggest that the main strategy for the validation of such models (and hence the justification for their use) still heavily relies on the comparison with historical observations. In this short paper, the author suggests that such a comparison alone is insufficient and that global sensitivity analysis provides additional possibilities for model evaluation to ensure greater transparency and better robustness of model-based analyses. Global sensitivity analysis can be used to demonstrate that the parameters defining intervention options (such as land use choices) adequately control the model output (even under potential future conditions); it can be used to understand the robustness of model outputs to input uncertainties over different projection horizons, the relevance of model assumptions, and how modelled environmental processes change with climatic boundary conditions. Such additional model evaluation would strengthen the stakeholder confidence in model projections and therefore into the adaptation strategies derived with the help of these model outputs.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
Introduction
Human activity has become a geologic-scale force, changing landscape and climate at increasing rates in our effort to supply societies growing demand for water, energy and food. A fundamental scientific and societal question of our time is: how will water, energy and biogeochemical cycles be altered by this activity, and when and where will critical thresholds be reached (Gleeson et al. 2020)? This insight, if available at the local or regional scale, is needed to guide decision- and policy-makers in the development of adaptation strategies and in designing infrastructure solutions suitable for future conditions (Barron 2009).
To develop adaptation strategies, impact models are needed to translate climate signals into hydrological, ecological or other decision-relevant variables to understand potential implications of future climate for security issues related to water (droughts and floods), food, energy and health. An important question when using such impact models is how they have been evaluated regarding their ability to perform their task adequately (Wagener et al. 2010)? Most often, the task of addressing this question is referred to as model validation. In a recent review of the validation of resource management models for a wide range of uses including scenario modelling, Eker et al. (2018) found that data-based strategies to model validation still prevail. The use of historical observations, to show that a model can reproduce observed system responses, remains the main approach to demonstrate that a model is a valid representation of reality. The use of the term validation itself has been criticized, because it is overpromising in the sense that it suggests that a model has been established as being true, rather than adequate for the task at hand (Oreskes et al. 1994). Therefore the author uses the term evaluation in this short paper to suggest that we can ever only achieve an incomplete and conditional assessment of a model’s suitability.
Any approach to evaluation suffers from multiple problems. First, the future might be significantly different from the past and demonstrating that a model is a realistic representation of the past system does not necessarily guarantee that it will reflect the future system. Many studies have therefore tried to create some type of resampling of the past to better reflect future conditions during model calibration/evaluation (e.g. Fowler et al. 2018; Singh et al. 2013), though this is only possible within limits. For example, even significant drought periods in the past will not fully reflect the combination of atmospheric, societal, land use and other conditions of the future. In some cases, modellers, therefore, prefer to run their models (especially across large domains) in uncalibrated mode, thus relying on the models’ physical realism. This, however, regularly leaves significant performance gaps between models and observed behaviour—a discrepancy that is typically not propagated into the assessment of future model projections. Second, comparing the model to historical data can ignore the intended use of the model, which one might expect to be the main driver of the evaluation strategy. Klemeš (1986) in his seminal paper introduces multiple ideas for validation strategies in relation to intended model use, e.g. related to modelling land use change. These ideas are still rarely fully implemented. Focusing on fitting historical data is also emphasising model performance, rather than model robustness in the presence of unavoidable uncertainties. And third, comparing the model to historical data might also ignore the manner in which stakeholders gain trust in model predictions, especially related to modelling change (Eker et al. 2018). Stakeholders might, for example, care strongly whether the model structure reflects an understanding of the real-world system that is consistent with their own (Mahmoud et al. 2009).
In this brief paper, how the use of global sensitivity analysis can be beneficial in enabling model evaluation elements that are complementary to assessing the model fit to historical observations would be highlighted (Wagener and Pianosi 2019). It is important to stress that the comparison of historical data is not useful, rather that it is insufficient. So, in this paper, we will briefly discuss using three examples from previously published studies and conclude with some general remarks and suggestions.
Standard Evaluation Questions We Can (and Should) Ask Using Global Sensitivity Analysis
Do the Parameters That Reflect Possible Intervention Levers Adequately Control the Model Output?
A key role of impact models is to create causal links between cause and effect variables, especially in the context of developing adaptation strategies. We might, for example, want to understand how much land use choices such as deforestation/reforestation control the level of downstream flooding under future climate conditions, or we might want to know the level of influence of human activities such as groundwater pumping on the overall drought risk under potential future warming scenarios. We, therefore, have to demonstrate that the parameters reflecting these intervention levers (such as those describing land use or human activities) actually exert an adequate control on the model output consistent with our current understanding.
For example, Butler et al. (2014) performed a comprehensive variance-based sensitivity analysis of a doubled-CO2 stabilization policy scenario generated by the globally aggregated dynamic integrated model of climate and the economy (DICE) (Fig. 5.1). The authors identified dominant processes by quantifying high sensitivities in model parameters relating to climate sensitivity, global participation in abatement and the cost of lower-emission energy sources. More importantly, in the context of this short paper, the authors did not find relevant sensitivities to other parameters such as those related to land use, that one might have expected to exert a stronger influence than the model shows. This result might suggest that certain intervention strategies cannot be assessed using the model in this particular example.
Are Dominant Uncertainties Changing Along the Projection Timeline?
It is further relevant to understand which uncertainties dominate the model output, especially over long time periods where levels of uncertainty might change considerably. Le Cozannet et al. (2015) used time-varying global sensitivity analysis to determine the factors that most strongly control the vulnerability of coastal flood defences over time (Fig. 5.2). They found that—for their question—global climate change scenarios only matter for long-term planning while local factors such as near-shore coastal bathymetry reflected in the wave setup parameter dominated in the short and mid-term (~over the next 50 years). The authors claim that wave setup uncertainty is often neglected in coastal hazard assessments studies. Global sensitivity analysis reveals that failing to incorporate the uncertainty in this process may invalidate conclusions and may lead to an overestimation of the effects of other drivers at least for short and mid-term planning periods. An assessment of the robustness of the model projections to input uncertainties thus has to consider the time-varying influence of these uncertainties.
Are Dominant Modelled Processes Changing with Climate?
And finally, how strongly different modelled environmental processes control the output of adaptation models can vary strongly with climate or other boundary conditions. Models behave differently depending on the climatic boundary conditions they are applied in, regardless of the level of physics the model is based on (Rosero et al. 2010). Figure 5.3 shows some results of a study by van Werkhoven et al. (2008) who tested the sensitivity of a model’s output (streamflow) to the parameters of a lumped rainfall-runoff model across 12 US catchments with very different climatic boundary conditions. The authors found (for both high flow and low flow conditions) that the controlling parameters varied considerably across climatic gradients. They further found that the spatial variability in sensitivity across catchments was similar to that observed within catchments when assessed across wet and dry years. This result suggests that for climate change projections, parameters (processes) that control the model behaviour for the historical period will likely differ from those that control the model output under new climatic boundary conditions. Global sensitivity analysis can provide insight into the degree of such changing model behaviour if a model is tested along climatic gradients.
Conclusions and Recommendations
Validation of impact models—an important task for developing adaptation strategies to climate change and for gaining stakeholder confidence—cannot be based on assessing a model’s fit to historical data alone, even though such assessment can clearly play a role in establishing confidence in a model (Fowler et al. 2018; Eker et al. 2018). It is further important that evaluation strategies are linked to the intended model use (Klemes 1986). In this context, global sensitivity analysis is a valuable tool to complement any data- and performance-based validation strategy since it allows us to make the model and its simulations significantly more transparent. It is important to stress here that sensitivity analysis can be applied regardless of whether observations of the system response are available or not, thus making it very suitable for understanding the behaviour of models under potential future conditions (Wagener and Pianosi 2019). Possible questions for sensitivity analysis are:
Do the model parameters that are linked to potential adaptation strategies (e.g. via land use choices) exert expected levels of control on the modelled output?
Which uncertainties are likely to dominate the model output during the relevant assessment period?
Which modelled processes are likely to dominate the model output under the projected climatic conditions (rather than under the conditions for which historical observations are available) and are we confident in those estimated parameters?
Are model projections (and more importantly subsequent decisions) robust to input uncertainties?
How sensitive are the model projections to model assumptions?
Global sensitivity analysis can provide a valuable additional component to strengthen our confidence as well as the confidence of stakeholders in climate change impact models (Wagener and Pianosi 2019; Saltelli et al. 2020). Recent studies have further demonstrated that such sensitivity analysis can be performed even on highly complex models (Maples et al. 2020) or on those covering a global domain (Reinecke et al. 2019).
References
Barron EJ (2009) Beyond climate science. Science 326:643
Butler MP, Reed PM, Fisher-Vanden K, Keller K, Wagener T (2014) Inaction and climate stabilization uncertainties lead to severe economic risks. Clim Change 127:463–474. https://doi.org/10.1007/s10584-014-1283-0
Eker S, Rovenskaya E, Obersteiner M, Langan S (2018) Practice and perspectives in the validation of resource management models. Nat Commun 9:5359
Fowler K et al (2018) Simulating runoff under changing climatic conditions: a framework for model improvement. Water Resour Res 54(12):9812–9983
Gleeson T et al (2020) Illuminating water cycle modifications and Earth System resilience in the Anthropocene. Water Resour Res 56(4):e2019WR024957
Klemeš V (1986) Operational testing of hydrological simulation models. Hydrol Sci J 31(1):13–24
Le Cozannet G, Rohmer J, Cazenave A, Idier D, van de Wal R, de Winter R, Pedreros R, Balouin Y, Vinchon C, Oliveros C (2015) Evaluating uncertainties of future marine flooding occurrence as sealevel rises. Environ Model Softw 73:44–56
Mahmoud M et al (2009) A formal framework for scenario development to support environmental decision-making. Environ Model Softw 24:798–808
Maples S, Foglia L, Fogg GE, Maxwell RM (2020) Sensitivity of hydrologic and geologic parameters on recharge processes in a highly-heterogeneous, Semi-confined aquifer system. Hydrol Earth Syst Sci https://doi.org/10.5194/hess-24-2437-2020.
Oreskes N, Shrader-Frechette K, Belitz K (1994) Verification, validation, and confirmation of numerical models in the earth sciences. Science 263(5147):641–646
Reinecke R et al (2019) Spatially distributed sensitivity of simulated global groundwater heads and flows to hydraulic conductivity, groundwater recharge, and surface water body parameterization. Hydrol Earth Syst Sci 23(11)
Rosero E, Yang Z-L, Wagener T, Gulden LE, Yatheendradas S, Niu G-Y (2010) Quantifying parameter sensitivity, interaction and transferability in hydrologically enhanced versions of Noah-LSM over transition zones. J Geophys Res 115:D03106. https://doi.org/10.1029/2009JD012035
Saltelli et al (2020) Five ways to ensure that models serve society: a manifesto. Nature 582
Singh R, van Werkhoven K, Wagener T (2013) Climate change impacts in gauged and ungauged basins of the Olifants River, South Africa. Hydrol Sci J https://doi.org/10.1080/02626667.2013.819431
van Werkhoven K, Wagener T, Reed P, Tang Y (2008) Characterization of watershed model behavior across a hydroclimatic gradient. Water Resour Res 44:W01429. https://doi.org/10.1029/2007WR006271
Wagener T, Pianosi F (2019) What has global sensitivity analysis ever done for us? A systematic review to support scientific advancement and to inform policy-making in earth system modelling. Earth Sci Rev https://doi.org/10.1016/j.earscirev.2019.04.006
Wagener T et al (2010) The future of hydrology: an evolving science for a changing world. Water Resour Res 46:W05301
Acknowledgements
This work on sensitivity analysis was initially supported by the UK Natural Environment Research Council (NERC) through the Consortium on Risk in the Environment: Diagnostics, Integration, Benchmarking, Learning and Elicitation (CREDIBLE) [NE/J017450/1]. Further support comes from a Royal Society Wolfson Research Merit Award and from an Alexander von Humboldt Professorship awarded to TW.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2022 The Author(s)
About this paper
Cite this paper
Wagener, T. (2022). On the Evaluation of Climate Change Impact Models for Adaptation Decisions. In: Kondrup, C., et al. Climate Adaptation Modelling. Springer Climate. Springer, Cham. https://doi.org/10.1007/978-3-030-86211-4_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-86211-4_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86210-7
Online ISBN: 978-3-030-86211-4
eBook Packages: Earth and Environmental ScienceEarth and Environmental Science (R0)