A Tool for Rating the Value of Health Education Mobile Apps to Enhance Student Learning (MARuL): Development and Usability Study

Background: To realize the potential for mobile learning in clinical skills acquisition, medical students and their teachers should be able to evaluate the value of an app to support student learning of clinical skills. To our knowledge, there is currently no rubric for evaluation of quality or value that is specific for apps to support medical student learning. Such a rubric might assist students to be more confident in using apps to support their learning. Objective: The objective of this study was to develop an instrument that can be used by health professional educators to rate the value of a mobile app to support health professional student learning. Methods: Using the literature, we developed a list of potential criteria for the evaluation of educational app value, which were then refined with a student group using a modified nominal group technique. The refined list was organized into themes, and the initial rubric, Mobile App Rubric for Learning (MARuL, version 1), was developed. iOS and Android app stores were searched for clinical skills apps that met our inclusion criteria. After the 2 reviewers were trained and the item descriptions were refined (version 2), a random sample of 10 included apps, 5 for each mobile operating system, was reviewed. Interitem and interrater analyses and discussions with the reviewers resulted in refinement of MARuL to version 3. The reviewers completed a review of 41 clinical skills mobile apps, and a second round of interitem and interrater reliability testing was performed, leading to version 4 of the MARuL. Results: Students identified 28 items (from an initial set of 144 possible items) during the nominal group phase, and these were then grouped into 4 themes: teaching and learning, user centered, professional, and usability. Testing and refinement with reviewers reduced the list to 26 items. Internal consistency for MARuL was excellent (a=.96), and the interrater reliability as measured by the intraclass correlation coefficient (ICC) was good (ICC=0.66). Conclusions: MARuL offers a fast and user-friendly method for teachers to select valuable apps to enhance student learning.