A measure of ordinal concordance for the evaluation of University courses

In this paper we apply the s* statistic, aimed to measure the inter-rater agreement between observers in case of ordinal variables, to the evaluation of the quality of University courses. The objective is to measure the inter-rater agreement between students, along with their satisfaction, in order to verify the consistency of judgments expressed by independent observers. s* is a modification of a previously proposed index, which avoids the problem of paradoxes of Cohen's and Fleiss' kappa statistics. We present the s* index from both a descriptive and an inferential point of view. In particular, as far as statistical inference is concerned, we show that s* is a biased estimator of the inter-rater agreement in the population and, under the null hypothesis of inter-rater agreement by chance, s* is asymptotically normally distributed. (C) 2014 The Authors. Published by Elsevier B.V.This is an open access article under the CC BY- NC- ND license