Verify-Pro: A Framework for Server Authentication using Communication Protocol Dialects

Customizing program binary and communication features is a commonly adopted strategy to counter network security threats like session hijacking, context confusion, and impersonation attacks. A potential attacker may have enough time to launch an attack targeting these vulnerabilities by rerouting the target request to a malicious server or hijacking the traffic. This paper presents a novel system Verify-Pro, a framework for server authentication using communication protocol dialects by customizing the communication features, enforcing continuous authentication, detecting the adversary, and preventing sensitive information leakage. Specifically, we leverage a machine learning approach (pre-trained neural network model) on both client and server machines to trigger a specific dialect that dynamically changes for each request (e.g., get filename in FTP). Then, a decision tree algorithm is developed to automatically detect the adversary and terminate the entire session if the message is from an adversary. We implement a prototype of VerifyPro and evaluate its practicality on standard communication protocol: FTP (File Transfer Protocol) and present a case study of the internet of things protocol MQTT (Message Queuing Telemetry Transport). Our experimental results show that by sending misleading information through the message packets from an attacker at the application layer, it is possible for the recipient to identify if the sender is genuine or a spoofed one, with a negligible overhead of < 1%.