Design and testing of a mobile health application rating tool

Mobile health applications ("apps") have rapidly proliferated, yet their ability to improve outcomes for patients remains unclear. A validated tool that addresses apps' potentially important dimensions has not been available to patients and clinicians. The objective of this study was to develop and preliminarily assess a usable, valid, and open-source rating tool to objectively measure the risks and benefits of health apps. We accomplished this by using a Delphi process, where we constructed an app rating tool called THESIS that could promote informed app selection. We used a systematic process to select chronic disease apps with >= 4 stars and <4-stars and then rated them with THESIS to examine the tool's interrater reliability and internal consistency. We rated 211 apps, finding they performed fair overall (3.02 out of 5 [95% CI, 2.96-3.09]), but especially poorly for privacy/security (2.21 out of 5 [95% CI, 2.11-2.32]), interoperability (1.75 [95% CI, 1.59-1.91]), and availability in multiple languages (1.43 out of 5 [95% CI, 1.30-1.56]). Ratings using THESIS had fair interrater reliability (kappa = 0.3-0.6) and excellent scale reliability (a = 0.85). Correlation with traditional star ratings was low (r = 0.24), suggesting THESIS captures issues beyond general user acceptance. Preliminary testing of THESIS suggests apps that serve patients with chronic disease could perform much better, particularly in privacy/security and interoperability. THESIS warrants further testing and may guide software and policymakers to further improve app performance, so apps can more consistently improve patient outcomes.