A Comparison of Statistical Methods of Reliability Generalization Meta-Analysis with Real Datasets

The reliability is a property that all tests must exhibit. Since it is not generalizable from one sample to another, the meta-analysis of reliability generalization was developed. Currently, there are many statistical methods, but there is no agreement on the most suitable for this type of meta-analysis. This study compares 15 different statistical methods in three real datasets regarding the distribution of reliability coefficients, the estimation of average reliability, and the effect of moderator variables. The results indicated that distributions tend to achieved normality with the transformations. Discrepancies in estimates of average reliability were small but could lead to different substantive conclusions. Moderator variables yielded significant results depending on the method employed. The discussion revolves around how this discrepancy between methods may influence on specific applications of reliability generalization studies.