Use of a Mobile App for the Process Evaluation of an Intervention in Health Care: Development and Usability Study

Background: Process evaluation measures the context in which an outcome was or was not achieved through the ongoing monitoring of operations. Mobile apps are a potentially less burdensome tool for collecting these metrics in real time from participants. Research-driven apps are not always developed while paying attention to their usability for target users. Usability testing uncovers gaps in researchers', developers', and users' mental models of what an efficient, effective, and satisfying product looks like and facilitates design improvement. Models may vary by user demographics. Objective: This study describes the development of a mobile app for collecting process evaluation metrics in an intervention study with health care workers that uses feedback at multiple stages to refine the app design, quantify usage based on workers' overall adoption of the app and the app's specific function, and compare the demographic and job characteristics of end users. Methods: An app was developed to evaluate the Center for Promotion of Health in the New England Workplace Healthy Workplace Participatory Program, which trains teams to develop solutions for workforce health obstacles. Labor-management health and safety committee members, program champions, and managers were invited to use the app. An accompanying website was available for team facilitators. The app's 4 functions were meeting creation, postmeeting surveys, project time logs, and chat messages. Google Analytics recorded screen time. Two stages of pilot tests assessed functionality and usability across different device software, hardware, and platforms. In stage 1, student testers assessed the first functional prototype by performing task scenarios expected from end users. Feedback was used to fix issues and inform further development. In stage 2, the app was offered to all study participants; volunteers completed task scenarios and provided feedback at deployment. End user data for 18 months after deployment were summarized and compared by user characteristics. Results: In stage 1, functionality problems were documented and fixed. The System Usability Scale scores from 7 student testers corresponded to good usability (mobile app=72.9; website=72.5), whereas 15 end users rated usability as ok (mobile app=64.7; website=62.5). Predominant usability themes from student testers were flexibility and efficiency and visibility of system status; end users prioritized flexibility andefficiency and recognition rather than recall. Both student testers and end users suggested useful features that would have resulted in the large-scale restructuring of the back end; these were considered for their benefits versus cost. In stage 2, the median total use time over 18 months was 10.9 minutes (IQR 23.8) and 14.5 visits (IQR 12.5). There were no observable patterns in use by demographic characteristics. Conclusions: Occupational health researchers developing a mobile app should budget for early and iterative testing to find and fix problems or usability issues, which can increase eventual product use and prevent potential gaps in data.