Teacher quality differences between teacher preparation programs: How big? How reliable? Which programs are different?

Sixteen US states have begun to hold teacher preparation programs (TPPs) accountable for teacher quality, where quality is estimated by teacher value-added to student test scores. Yet it is not easy to identify TPPs whose teachers are substantially better or worse than average. True teacher quality differences between TPPs are small; estimated differences are not very reliable; and when many TPPs are compared, multiple comparisons increase the danger of misclassifying ordinary TPPs as good or bad. Using a large and diverse dataset from Texas, we evaluate statistical methods for estimating teacher quality differences between TPPs. The most convincing estimates come from a value-added model where confidence intervals are widened by the inclusion of teacher random effects (or teacher clustering in large TPPs) and further widened by the Bonferroni correction for multiple comparisons. Using these confidence intervals, it is rarely possible to tell which TPPs, if any, are better or worse than average. The potential benefits of TPP accountability may be too small to balance the risk that a proliferation of noisy TPP estimates will encourage arbitrary and ineffective policy actions. (C) 2016 Elsevier Ltd. All rights reserved.