Insufficiency of traditional statistical model validation for models with many tentative regressors

The most used statistical model validation methods are constructed to detect correlation between residuals from the model and regressors not included in the model. If there is statistical confidence in the correlation then the model is rejected. In this contribution it is shown that such tests are useless for problems where the number of possible regressors is high, e.g., in block box identification of nonlinear systems and when FIR models used to approximate infinite impulse responses. Due to the bias-variance trade-off it is not possible to include all regressors. The purpose of the statistical test should be to decide which, and how many regressors should be included in the model. The significance for the last included regressors will typically be small and it is shown that the statistical test will make the wrong decision upon these last regressors. The result holds for all significance levels of the statistical test.