Formulating Parallel Supervised Machine Learning Designs For Anomaly-Based Network Intrusion Detection in Resource Constrained Use Cases

One of the main problems with using supervised machine learning for anomaly-based Network Intrusion Detection (NID) in large cyber networks is the massive and dynamic data sets used in training and the computational overhead they induce in the training phase. In resource-constrained situations, there is a lack of powerful machines that would be traditionally used to deal with a high computational load. We focus on parallel processing designs that work in such resource-constrained use cases by lowering the computational overhead of supervised machine learning for anomaly-based NID, specifically for Mini-batch gradient descent. We avoid the black-box approach of traditional parallelization frameworks and allow the user to maximize the utilization of their scarce computational resources by granting greater control over parallelization while leveraging key aspects of the optimization algorithms being implemented. To demonstrate this, we implemented initial data and model parallel based designs, using the Compute Unified Device Architecture (CUDA) and Message Passing Interface (MPI) libraries, aimed at maximizing the use of a limited number of CPUs and GPUs. We conducted an initial comparative performance study using a large real-world network intrusion dataset called the KDD cup 1999; our results demonstrate up to 8.5 times more epochs per second using just 1 GPU (4000 threads) and up to 37 times faster convergence using just 1 compute node (7 cores) when compared to a serial approach.