Using simulation to evaluate prediction techniques

The need for accurate software prediction systems increases as software becomes much larger and more complex. A variety of techniques have been proposed, however, none has proved consistently accurate and there is still much uncertainty as to what technique suits which type of prediction problem. We believe that the underlying characteristics - size, number of features, type of distribution, etc. - of the dataset influence the choice of the prediction system to be used. In previous work, it has proved difficult to obtain significant results over small datasets. Consequently we required large validation datasets, moreover, we wished to control the characteristics of such datasets in order to systematically explore the relationship between accuracy, choice of prediction system and dataset characteristic. Our solution has been to simulate data allowing both control and the possibility of large (1000) validation cases. In this paper we compared regression, rule induction and nearest neighbour (a form of case based reasoning). The results suggest that there are significant differences depending upon the characteristics of the dataset. Consequently researchers should consider prediction context when evaluating competing prediction systems. We also observed that the more "messy" the data and the more complex the relationship with the dependent variable the more variability in the results. This became apparent since we sampled two different training sets from each simulated population of data. In the more complex cases we observed significantly different results depending upon the training set. This suggests that researchers will need to exercise caution when comparing different approaches and utilise procedures such as bootstrapping in order to generate multiple samples for training purposes.