Declarative platform for high-performance network traffic analytics

This paper presents Scalanytics, a declarative platform that supports high-performance application layer analysis of network traffic. Scalanytics uses (1) stateful network packet processing techniques for extracting application layer data from network packets, (2) a declarative rule-based language called Analog for compactly specifying analysis pipelines from reusable modules, and (3) a task-stealing architecture for processing network packets at high throughput within these pipelines, by leveraging multi-core processing capabilities in a load-balanced manner without the need for explicit performance profiling. In a cluster of machines, Scalanytics further improves throughput through the use of a consistent-hashing based load partitioning strategy. Our evaluation on a 16-core machine demonstrate that Scalanytics achieves up to 11.4×\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times $$\end{document} improvement in throughput compared with the best uniprocessor implementation. Moreover, Scalanytics outperforms the Bro intrusion detection system by an order of magnitude when used for analyzing SMTP traffic. We further observed increased throughput when running Scalanytics pipelines across multiple machines.