Developing improved metamodels by combining phenomenological reasoning with statistical methods

A metamodel is relatively small, simple model that approximates the "behavior" of a large, complex model. A common and superficially attractive-way to develop a metamodel is to generate "data" from a number of large-model runs and to then use off-the-shelf statistical methods without attempting to understand the model's internal workings. This paper describes research illuminating why it is important and fruitful, in some problems, to improve the quality of such metamodels by using various types of phenomenological knowledge. The benefits are sometimes mathematically subtle, but strategically important, as when one is dealing with a system that could fail if any of several critical components fail. Naive metamodels may fail to reflect the individual criticality of such components and may therefore be quite misleading if used for policy analysis. Naive metamodeling may also give very misleading results on the relative importance of inputs, thereby skewing resource-allocation decisions. By inserting an appropriate dose of theory, however, such problems can be greatly mitigated. Our work is intended to be a contribution to the emerging understanding of multiresolution, multiperspective modeling (MRMPM), as well as a contribution to interdisciplinary work combining virtues of statistical methodology with virtues of more theory-based work. Although the analysis we present is based on a particular experiment with a particular "large and complex model," we believe that the insights are more general.