Dynamic multi-objective sequence-wise recommendation framework via deep reinforcement learning

Sequence-wise recommendation, where recommend exercises to each student step by step, is one of the most exciting tasks in the field of intelligent tutoring systems (ITS). It is important to develop a personalized sequence-wise recommendation framework that immerses students in learning and helps them acquire as much necessary knowledge as possible, rather than merely focusing on providing non-mastered exercises, which is referred to optimize a single objective. However, due to the different knowledge levels of students and the large scale of exercise banks, it is difficult to generate a personalized exercise recommendation for each student. To fully exploit the multifaceted beneficial information collected from e-learning platforms, we design a dynamic multi-objective sequence-wise recommendation framework via deep reinforcement learning, i.e., DMoSwR-DRL, which automatically select the most suitable exercises for each student based on the well-designed domain-objective rewards. Within this framework, the interaction between students and exercises can be explicitly modeled by integrating the actor–critic network and the state representation component, which can greatly help the agent perform effective reinforcement learning. Specifically, we carefully design a state representation module with dynamic recurrent mechanism, which integrates concept information and exercise difficulty level, thus generating a continuous state representation of the student. Subsequently, a flexible reward function is designed to simultaneously optimize the four domain-specific objectives of difficulty, novelty, coverage, and diversity, providing the students with a trade-off sequence-wise recommendation. To set up the online evaluation, we test DMoSwR-DRL on a simulated environment which can model qualitative development of knowledge level and predicts their performance for a given exercise. Comprehensive experiments are conducted on four classical exercise-answer datasets, and the results show the effectiveness and advantages of DMoSwR-DRL in terms of recommendation quality.