Comment on "Rubric-based holistic review represents a change from traditional graduate admissions approaches in physics"

A recent paper evaluating a new rubric-based graduate admissions approach using generic methods tentatively suggested that its decisions differed noticeably from the previous approach in an unspecified way. Using prior knowledge that the often-stated specific goal was to open a path to increased diversity by reducing barriers to the admission of applicants with low undergraduate grade point averages and graduate record exam (GRE) scores allows simple statistical tests of changes in the distributions of the metrics. The simple tests confirm with good statistical confidence that the barrier-reduction changes were achieved. Nevertheless, the paper's argument that the de-emphasized tests and grades are not predictive of graduate outcomes is not supported by the prior literature. On technique, although a method used in some of the analyses for dropping data points before running the machine algorithm is likely to bias those results, it helps to clarify why models of the rubric-based system were only weakly predictive.