Fitting Statistical Models of Random Search in Simulation Studies

We consider optimization of expected system performance by random search. There are two sources of random variation in this process: (i) a search-induced variability because the expected performance of the system will vary randomly according to the alternatives randomly selected for examination, and (ii) a simulation induced variability, because there will be random error in estimating expected system performance from finite simulation runs. We show that, in altering the balance between these two sources of variability, three distinct forms of asymptotic behavior of the estimate of the optimal expected system performance are possible. The form of the asymptotic results shows that they may be not be easy to apply in practical work. As an alternative, a methodology for fitting a statistical model that accounts for both types of variability is suggested. This then allows the distributional properties of quantities of interest, like the optimum performance value and the best value obtained by the search, to be estimated by resampling and which also allows a test of goodness of fit of the model. Four numerical examples are given.