Defining and estimating the reliability of physician quality measures in hierarchical logistic regression models

In healthcare provider profiling, the reliability of a performance measure indicates whether observed differences in patient outcomes can be attributed to genuine differences in quality across providers. While reliability is easy to define, estimate, and interpret when the outcome of interest is continuous and a hierarchical linear model can be assumed, several different definitions and estimators of reliability are in use for performance measures based on binary outcomes. We compare these candidate definitions and estimators when a hierarchical logistic regression model is assumed for the binary outcome. Our simulations revealed important differences across reliability estimators, particularly when the number of patients per provider is low, as expected for physician quality measures. Using claims data on Medicare fee-for-service beneficiaries treated by physicians in Florida, we illustrate the practical implications by examining two process-of-care measures collected by the Centers for Medicare and Medicaid Services in the U.S. for physician quality reporting.