A new cost function for model fitting: the length-scale of dynamical consistency

This article introduces a cost function for model fitting that exploits geometrical properties of the models' dynamics to improve the estimation of uncertainty. The cost function can be thought of as an operational way of asking the following question: are the observations consistent with the dynamics of the model to within the precision granted by a prescribed level of uncertainty? Model parameters which allow for low levels of uncertainty while retaining consistency are taken to provide a better fit than parameters for which the level of uncertainty must be increased if consistency is to be preserved. The minimum level of uncertainty is the statistic which is returned by the cost function for a given set of parameters, similar to the way sums of squared errors is returned by a maximum likelihood function with Gaussian errors. Unlike maximum likelihood however, this output can be interpreted directly in physical (or biological) terms: it is the smallest length scale of the system variable (e. g. biomass) at which predictability can be maintained; that is, forecasts can be made within plus or minus this length scale. We term this cost function the 'length-scale of dynamical consistency' (LSDC). All techniques for model fitting must at some point, directly or indirectly, measure a discrepancy (that is, a distance) between model predictions and reality (the observations). A critical methodological step taken in this paper is that we are no longer measuring this distance in the Euclidean space of the observations, but rather in the (almost always) non-Euclidean space of the dynamics. The cost function is validated with a numerical experiment where the true dynamics are known and noise is artificially added. The experiment confirms that the technique is accurate with respect to converging on the true parameter values, but also that it provides additional information about parameter uncertainty compared to a reference least squares approach. In particular it clearly identifies a small region of parameter space with minimal length scale, the shape of which reflects the parameter confounding set up in the experiment, and for surrounding regions the length scale jumps sharply. In contrast the least squares approach displays a relatively flat solution surface. The cost function is also applied to a real data set, a series of catch and effort data from Australia's northern prawn fishery. The LSDC results and those of a reference maximum likelihood implementation differ in a non-trivial fashion and further work is needed to interpret this fully. While the theoretical framework is explained in some detail, the results presented here are preliminary in nature and numerical investigations are ongoing.