Vision Aided Beam Tracking and Frequency Handoff for mmWave Communications

Vulnerability to blockage and time-consuming beam tracking are two important issues yet to be solved in millimeter-wave (mmWave) communications systems. In this paper, we demonstrate stereo camera and LiDAR aided beam tracking and blockage prediction platforms for mmWave communications that do not cost in-band communications resources, e.g., time for pilot training and beam sweeping. In stereo camera aided platform, we perform beam tracking at a rate of 12 fps as well as frequency switching from mmWave to sub-6G right before the blockage happens. On the other side, LiDAR aided platform is mainly used to perform beam tracking under dark light environment, in which case the camera cannot accurately capture the vision information. It can be seen that the two platforms can establish the mmWave communications links with vision information only and can successfully predict the blockage.