MedicalCare: building and annotating an empathy-rich corpus

The importance of empathy in clinical settings has been widely accepted in the research community, and there have been numerous attempts at training clinical practitioners in empathic communication. Despite the advances in affective computing and automatic recognition and classification of emotions in discourse, there has been little research on how to characterize and model empathy in clinical settings. A corpus of essays was collected as a preliminary dataset for building an early stage linguistic model and measuring the efficacy of inter-annotator agreement on such a dataset. As annotated corpora have been popular resources for research on affective computing, in this study we build a text corpus named MedicalCare, and annotate it for empathic expressions using an iterative annotation process. We evaluated the annotation quality and the level of inter-annotator agreement over time, and found steady improvement in inter-annotator agreement on sentence labels as well as elaboration of the annotation guidelines. The average inter-rater agreement obtained over 370 essays annotated by four annotators is kappa = 0.65, and kappa = 0.82 between two meta-annotators. We also conducted text analyses of the annotated essays and found that the use of personal pronouns, negative emotion words and words about reassurance are correlated with empathic expressions.