Towards a Reliable Hierarchical Android Malware Detection Through Image-based CNN

The number of Android malicious applications keeps growing as time passes, even paving their way to official app markets. In recent years, a promising malware detection approach makes use of the compiled app source codes (dex), through convolutional neural networks (CNN) as an image classification task. Unfortunately, current proposals often rely on unrealistic datasets, focusing their detection on the malware families, while neglecting the detection of malware apps in the first place. In this paper, we propose a reliable and hierarchical Android malware detection through an image-based CNN scheme, implemented twofold. First, Android malware classification is performed in a hierarchically-structured local manner, initially identifying malware apps, then, their related family. Second, to ensure reliability and improve classification accuracy, only highly confident classified apps are reported, in a classification with reject option rationale. Experiments performed in a new dataset with over 26 thousand Android apps, divided into 29 malware families, compounding over 13 GB of app dex images, have shown that current image-based CNN for malware detection is unable to provide high detection accuracies. In contrast, our proposed model is able to reliably detect malware apps, improving the true-negative rates by up to 5.5%, and the average true-positive rate of the malware families of accepted apps by up to 12.7%, while rejecting only 10% of Android apps.