Comparing Many-facet Rasch Model and ANOVA model: Analysis of ratings of essays

The Many-facet Rasch Model (MFRM) and the analysis of variance model (ANOVA) were applied to ratings by 10 graduate students on 5-point scales of 5 aspects of summaries of 25 theses submitted in support of the bachelor's degree in educational psychology. Both models detect idiosyncratic ratings by using residual scores. The results indicated that the residuals in the 2 models were highly correlated, although there was some difference between the models in sensitivity to certain types of idiosyncrasy. A high correlation was also found in simulated data with varying degrees of model fit. In the actual data, the thesis-by-rater interaction was substantial; this was not taken into account in the preceding comparison. Application of the ANOVA model with that interaction led to detection of some idiosyncratic ratings that had been overlooked in both the Many-Facet Rasch Model and the main-effect ANOVA model. Flexibility in incorporating such interactions in the model, the number of raters needed for analysis, and the availability of software all favored the ANOVA model.