Background: Music as alternate engagement (MAE) can be used effectively to distract children during painful or anxiety-provoking medical procedures. For such interventions to be successful, it would seem important to assess the degree to which a child can attend to musical stimuli. Objective: The purposes of this study were as follows: (a) To establish construct validity by determining the extent to which the Music Attentiveness Screening Assessment (MASA) measures auditory attention; and (b) to gather evidence regarding MASA test-retest and inter-observer reliability. Methods: The Auditory Attention (AA) subtest from the NEPSY-II (NEPSY, Second Edition) and the two items from MASA were administered to a nonclinical sample of children (N = 50) aged 5 to 9 years. Results: There was a statistically significant proportion of AA score variance shared with MASA (both items), R-2 = .21, F(2, 47) = 6.34, p = .004. Test-retest reliability on the first MASA item was moderately high (Pearson r = .84) while on the second item it was lower (r = .63). Similarly, interobserver agreement was high for Item I (intraclass correlation coefficient [ICC] = .95) and lower for Item II (ICC = .71). Conclusions: Evidence suggests that MASA measures, at least in part, auditory attention. Despite this finding, a large proportion of unexplained variance remains. Furthermore, reliability estimates (test-retest and interobserver agreement) differ between both items. These findings are discussed with particular attention paid to the ways in which MASA should be revised and further study conducted.