Data-Driven Design Pattern Production: A Case Study on the ASSISTments Online Learning System

Recently, online learning systems such as cognitive tutors, online courses, and massive open online courses (MOOCS) increased in popularity in various domains. The design quality of online learning systems is difficult to maintain. Multiple stakeholders are involved (e.g., software developers, interaction designers, learning scientists, teachers), the system is complex, there are rapid changes in software, platforms (e.g., mobile, tablet, desktop) and learning subject content, and so forth. Many existing online learning systems collect a significant amount of data that describe students' learning gains and affective states, which are indirect measures of system quality. Analysis of online learning system data can uncover linkages between particular design choices made and student learning. In this paper, we describe the data-driven design pattern production (3D2P) methodology to prospect, mine, write, and evaluate design patterns for online learning systems from collected data. Design patterns are high quality solutions for recurring problems in a particular context. Patterns identified with 3D2P can guide the addition of new content and the modification of system designs to maintain the quality of online learning systems. A case study on the ASSISTments math online learning system is presented to demonstrate the 3D2P methodology and discuss its benefits and limitations.