Assessing climate service products with evaluation metrics: an application to decision support tools for climate change adaptation in the USA

As the field of climate services advances, there is increasing recognition of the need for evaluation of climate services. Existing research largely includes detailed evaluations and discussions of a single or small number of services or broad-scale sampling activities that record basic information on a very large number of services. In this work, we propose an intermediate approach to evaluating existing climate services that includes evaluation metrics with definitions for high, medium, and low performance based on a previously proposed evaluation framework. The evaluation metrics characterize the type of information provided by a climate service, the manner in which it is communicated, and the degree to which the tool's development background is transparently presented. We use the proposed metrics to evaluate a sample of 20 US-based online decision support tools aimed at supporting climate change adaptation. We find that most tools are quite thorough in their presentation of detailed climate information and inclusion of explanatory material. However, few tools provide much discussion of the specific decisions that the tool aims to support or the role that climate information can play in these decisions. This work contributes an approach for intermediate-scale climate service evaluation that provides a middle ground between detailed case studies and broad-scale sampling. This can complement existing evaluation research and also highlight potential issues and gaps in climate service research and evaluation.