Real-time audio-visual localization of user using microphone array and vision camera

In home environments, demands for a robot to serve a user are on the increase, such as cleaning rooms, bringing something to the user, and so on. To achieve these tasks, it is essential for developing a natural way of Human-Robot Interaction (HRI). One of the most natural ways is that the robot approaches the user to do some tasks after recognizing the user's call and localizing its position. In this case, user localization becomes a key technology. In this paper, we propose a novel audio-visual user localization system. It consists of a microphone array with eight sensors and a video camera. Estimating calling direction is achieved by the spectral subtraction of the spatial spectra. In particular, a novel beam-forming method is proposed to suppress the non-stationary audio noises where they always occur in a real world. Furthermore, a robust method for face detection is proposed to double-check the user based on an Adaboost classifier. It is improved to reduce the false alarms remarkably through a new post-processing on face candidates. Successful results in a real home environment show its efficacy and feasibility. The implementation issues, limitations, and their possible solutions are also discussed.