Device Risk Analysis Protocol for SMS-Based OTP Authentication

Two-factor authentication (2FA) is widely recognized as a secure authentication method. Despite the availability of multiple authentication methods, SMS one-time password (OTP) remains popular. However, SMS OTP is vulnerable to several attacks that pose a significant threat to the authentication process. Due to the risk of attacks, particularly those based on social engineering and malware related to endpoint compromise, the National Institute of Standards and Technology (NIST) has removed SMS OTP as a recommended delivery channel. This paper analyses two different variants of passive and active malware attacks on SMS OTP for Android mobile devices. In response to the identified threats, a risk assessment protocol is proposed. This protocol includes a malware detection algorithm to assess device risk and determine whether SMS OTP can be used for user authentication. The security level of the authentication process depends on the user's specific device. The proposed malware detection algorithm was tested on publicly available applications provided by users participating in the research. Two application datasets were scanned during the research. The first dataset consisted of 520 applications available on the Play Store, and the second dataset consisted of 1,200 applications provided by users who participated in the research.