The reliability of assessment criteria for undergraduate medical students' communication skills portfolios: the Nottingham experience

INTRODUCTION Some educators have argued that portfolios should not be assessed summatively because there is little evidence supporting the reliability of their assessment. This study aims to determine the reliability of assessment criteria used for a portfolio at the University of Nottingham. METHODS Two independent analysts assessed a random sample of portfolios (n = 100, 49.5%) using criterion-referenced assessment. Students' performances were examined against subjective items in five areas: 1) portfolio structure, 2) level of critical reflection, 3) level of skills development, 4) use of documentary evidence, and 5) use of relevant literature. These subjective judgements were later converted into quantitative scales ranging from 0 to 3 so that interrater reliability could be established. The level of agreement between the two analysts for the total percentage score was established using an intraclass correlation coefficient and for the individual items using weighted kappa coefficients. RESULTS The level of agreement between the two raters for the total percentage score was 0.771 (95% CI = 0.678, 0.840), as measured by an intraclass correlation coefficient. The levels of agreement between the two raters for the individual items of the assessment criteria ranged from kappa = 0.359 (item 3) to kappa = 0.693 (item 4). DISCUSSION This study provides some support for the summative assessment of portfolios. The findings suggest that discussion and negotiation between independent assessors can enhance the reliability of assessment criteria. Therefore, medical educators are encouraged to use such procedures in the summative assessment of portfolios.