MetaSketch: Wireless Semantic Segmentation by Reconfigurable Intelligent Surfaces

Semantic segmentation is a process of partitioning an image into segments for recognizing regions of humans and objects, which can be widely applied in scenarios such as healthcare and safety monitoring. To avoid privacy violation, using radio frequency (RF) signals instead of photos for semantic segmentation has gained increasing attention. However, traditional human and object recognition by using RF signals is a passive signal collection and analysis process without changing the radio environment. The recognition accuracy is restricted significantly by unwanted multi-path fading, and/or the limited number of independent channels between RF transceivers. This paper introduces MetaSketch, a novel RF-sensing system that performs semantic recognition and segmentation for humans and objects by making the radio environment reconfigurable. A metamaterial-based reconfigurable intelligent surface is incorporated to diversify the information carried by RF signals. Using compressive sensing techniques, MetaSketch reconstructs a point cloud consisting of the reflection coefficients of humans and objects at different spatial points, and recognizes the semantic meaning of the points by using symmetric multilayer perceptron groups. Our evaluation results show that MetaSketch is capable of generating favorable radio environments, extracting exact point clouds, and labeling the semantic meaning of the points with an average error rate of less than 1% in an indoor space.