Network security situational awareness and early warning architecture based on big data

As the scale of the internet continues to expand and complex attack methods such as Advanced Persistent Threats (APTs) emerge, traditional Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) face high rates of false positives and false negatives, creating an urgent need for a more robust network protection mechanism. To address these challenges, this paper proposes a big data-driven network security situational awareness and early warning architecture. The implementation steps include: building a data storage system using Hadoop Distributed File System (HDFS), Hive, and HBase, with HBase responsible for fast retrieval. MapReduce is used for large-scale data processing, combined with data mining techniques and Long Short-Term Memory (LSTM) networks, and Apache Mahout is employed to encapsulate traditional algorithms. A flexible situational awareness platform is designed, integrating various security devices and covering information integration, data analysis, multidimensional visualization, and warning processing. Data is stored in HDFS, Hive, and HBase, analyzed using LSTM networks, and real-time information is correlated to predict threats and generate warnings. This big data-driven network security architecture aims to enhance protection capabilities and response speed. Comparative evaluation with traditional protection systems showed that the big data-based security system increased network port traffic by approximately 50%, reduced memory usage by 36%, significantly shortened response time, and improved the security posture score by 0.19. The big data system effectively isolates external malicious information, ensuring public information security and reducing losses. This study provides significant progress in the field of network protection by offering a more robust and proactive defense mechanism against emerging threats, ultimately reducing potential risks and enhancing overall network security.