Analysis of simulation results using statistical models

Simulation results for the comparison of estimators of interest are usually reported in tables or plots. However, if the simulations are conducted under various conditions for many estimators, the comparison can be difficult to be made with tables or plots. Furthermore, for algorithms that take a long time to run, the number of iterations of the simulation is costly to to be increased. The analysis of simulation results using regression models allows us to compare the estimators more systematically and effectively. Since variances in performance measures may vary depending on the simulation conditions and estimators, the heteroscedasticity of the error term should be allowed in the regression model. And multiple comparisons should be made because multiple estimators should be compared simultaneously. We introduce background theories of heteroscedasticity and multiple comparisons in the context of analyzing simulation results. We also present a concrete example.