Quality assessment of digital voice assistants on information provided in eating disorders and coexisting depression

INTRODUCTION: Patients with eating disorders and coexisting depression often rely on the Internet, and digital voice assistants (VAs) as methods of searching for health-related information regarding their conditions. However, the quality of information provided by VAs is questionable. We evaluated the quality of information on eating disorders and coexisting depression from 4 commonly-used VAs (Google Assistant [Google LLC, Menlo Park, CA, USA]; Siri [Apple Inc, Cupertino, CA, USA]; Cortana [Microsoft Corporation, Albuquerque, NM, USA]; Bixby [Samsung Electronics Co., Ltd, Suwon, South Korea]) and Google Search (Google LLC, USA). EVIDENCE ACQUISITION: Forty-four questions on eating disorders and coexisting depression were evaluated. Their responses were evaluated by two raters for accuracy (score: 2), source expertise (score: 1), underlying references cited (score: 2) and comprehensiveness (score: 2) using a scoring matrix (score: 8). Descriptive statistics and odds ratios were used for analysis. Cohen Kappa was used to measure inter-rater agreement. EVIDENCE SYNTHESIS: Cortana (mean=5.23 +/- 2.01) and Siri (mean=4.42 +/- 2.50) scored the highest and lowest for overall quality, respectively. Cortana (41/44, 93.2%) and Bixby (32/44, 72.7%) provided the most and least number of relevant sources (41/44, 93.2% versus 32/44, 72.7%, P<0.0001), and the highest and lowest mean accuracy scores (1.82 +/- 0.54 versus 1.43 +/- 0.89, P=0.0016) respectively. Bixby was the most reliable in terms of source expertise (mean=0.43 +/- 0.50) and underlying references cited (mean=0.93 +/- 0.50). Google Search scored the highest in terms of comprehensiveness, while Siri performed the worst for comprehensiveness, source expertise and underlying references cited. CONCLUSIONS: Most of the sources provided by the VAs were accurate and comprehensive, but not as reliable. Patients should be cautious when using VAs to search for information on eating disorders and coexisting depression.