Experimental concepts for testing probabilistic earthquake forecasting and seismic hazard models

To account for the randomness (aleatory variability) and limited knowledge (epistemic uncertainty) of earthquake processes, we must formulate and test seismic hazard models using the concepts of probability. Probabilistic seismic hazard analysis (PSHA) gives the chance that a specified seismic intensity will be exceeded at a particular site during some future time window. Because knowledge of that chance is uncertain, a complete PSHA estimate of probabilities must itself be expressed as a probability distribution, often quantified using expert opinion. The conceptual objections to PSHA stem primarily from the unsatisfactory treatment of the aleatory-epistemic duality in probabilistic inference and questions regarding the inherent testability of subjective models. We have proposed an augmented probabilistic framework that resolves these issues by adding a third level of uncertainty, which we term 'ontological error' (PNAS, 111, 11973, 2014). A complete probabilistic model is sufficient to pose an ontological null hypothesis, which states that the aleatory representation of future earthquake activity-the seismic behaviour of the real Earth-samples the probability distribution of aleatory representations that describe the model's epistemic uncertainty. We can test, and possibly reject, the ontological null hypothesis by employing an 'experimental concept' that identifies collections of data, observed and not yet observed, judged to be exchangeable (i.e. with joint probability distributions invariant to data ordering) when conditioned on a set of explanatory variables. These conditional exchangeability judgments specify data sequences with well-defined frequencies. Bayesian models that predict such long-run frequencies can thus be tested for ontological error by frequentist methods, for example using P-values. We describe examples of experimental concepts pertinent to the ontological testing of complete probabilistic earthquake forecasting models. The examples, which include both simple and compound exceedance testing, illustrate how test distributions can be derived from the de Finetti representation and its generalizations, the use of hierarchical statistical models to express the various layers of aleatory variability, and the importance of the 'Principle of Complete Ignorance' in characterizing ontological errors. We also show how differing experimental concepts can probe different model features and explore how experimental concepts can themselves be tested.