Do NoT Open (DOT): A Unified Generic and Specialized Models for Detecting Malicious Email Attachments

In this paper, we propose - DOT - a hybrid analysis approach designed for the detection and classification of malicious files. We have developed both a unified single model and specialized models tailored to various file extensions. Our solutions leverage byte-level content analysis to identify malicious elements within documents, along with n-gram analysis. The uniqueness of DOT lies in its ability to significantly reduce computational overhead. We achieve this by employing Rolling Encoder Hashing, which shortens bytecode sequences, making them compatible with state-of-the-art sequence models like Recurrent Neural Networks (RNNs). Additionally, we have created a static analysis-based generic model capable of working with a variety of file types, including.doc,.docx,.xls,.xlsx,.pdf, and more. This model can be efficiently deployed in realworld scenarios. Furthermore, we have developed specialized models for different file types, which are enhanced versions of the generic architecture, streamlining complex maintenance procedures. Another key innovation and novelty of DOT lies in exactly locating the portion of content in the byte code that could contain malicious code, to help security analysts make the binary code analysis more efficient. We conducted extensive experiments using a dataset recently made available by sources like VirusShare, Contagio, and others, specifically intended for academic research. Our dataset comprises a substantial collection of over 156,000 documents, encompassing both malicious and benign files of the most hazardous types observed in recent years. Our findings reveal impressive results, with a unified single model achieving a 91.43% accuracy in distinguishing between benign and malicious documents. Furthermore, specialized models tailored to specific file types exhibit even higher accuracy rates: 96.13% for.doc files, 97.85% for.docx files, 92.62% for.xls files, 97.02% for.xlsx files, and 94.11% for.pdf files, respectively and with a very low false positive rate.