WiVi: WiFi-Video Cross-Modal Fusion based Multi-Path Gait Recognition System

WiFi-based gait recognition is an attractive method for device-free user identification, but path-sensitive Channel State Information (CSI) hinders its application in multi-path environments, which exacerbates sampling and deployment costs (i.e., large number of samples and multiple specially placed devices). On the other hand, although video-based ideal CSI generation is promising for dramatically reducing samples, the missing environment-related information in the ideal CSI makes it unsuitable for general indoor scenarios with multiple walking paths. In this paper, we propose WiVi, a WiFi-video cross-modal fusion based multi-path gait recognition system which needs fewer samples and fewer devices simultaneously. When the subject walks naturally in the room, we determine whether he/she is walking on the predefined judgment paths with a K-Nearest Neighbors (KNN) classifier working on the WiFi-based human localization results. For each judgment path, we generate the ideal CSI through video-based simulation to decrease the number of needed samples, and adopt two separated neural networks (NNs) to fulfill environment-aware comparison among the ideal and measured CSIs. The first network is supervised by measured CSI samples, and learns to obtain the semi-ideal CSI features which contain the room-specific 'accent', i.e., the long-term environment influence normally caused by room layout. The second network is trained for similarity evaluation between the semi-ideal and measured features, with the existence of short-term environment influence such as channel variation or noises. We implement the prototype system and conduct extensive experiments to evaluate the performance. Experimental results show that WiVi's recognition accuracy ranges from 85.4% for a 6-person group to 98.0% for a 3-person group. As compared with single-path gait recognition systems, we achieve average 113.8% performance improvement. As compared with the other multi-path gait recognition systems, we achieve similar or even better performance with needed samples being reduced by 57.1-93.7%