A Machine Learning Grading System Using Chatbots

The ability to provide students with timely and accurate feedback is critical to learning. However, grading written essays is demanding, and can be challenging to conduct in large classes. We explore an automated grading system involving the use of a Chatbot that asks students questions, requiring written responses. We implemented unsupervised machine learning techniques for the task of automated grading and conducted an experiment to assess the performance of the Chatbot as compared to human grading. The experiment involved posting questions to 15 students, requiring short written answers. To analyse the performance of the Chatbot, we used a combination of term-frequency inverse-document function (tfidf) with cosine Euclidean distance, and online semantic text analytics (Dandelion API), trained with neural networks on a large bank of questions and answer dataset. We then used Cohen's kappa agreement. The result shows a good inter-rater agreement level between the automated grading and the human instructor. The work presented in the paper presents open up opportunities for using Chatbots in providing automated assessment and at the same time fosters engagement with students.