Management of scoring sessions in alternative assessment: the computer-assisted scoring approach

We describe a computer-assisted scoring approach in educational assessment. In this approach, scores are captured and analyzed as scoring takes place; information on scoring quality is used to provide immediate feedback to raters and make timely re-calibration and dismissal decisions. We present a conceptual model for computer-assisted scoring and describe how we used this approach to manage the scoring sessions of an assessment for teacher certification. We found that computer-assisted scoring: (a) allowed us to provide immediate feedback to raters about their scaring quality and make accurate recalibration and dismissal decisions; and (b) did not affect the dependability of the scores or the flow of the scoring sessions. We also confirmed that, with appropriate software, raters can be trained readily to score complex performance with the aid of computers even when they have no prior experience with computers. The conceptual model allowed us to identify how close our scoring sessions were to optimal efficiency. (C) 1999 Elsevier Science Ltd. All rights reserved.