Object Tracking with a Fusion of Event-Based Camera and Frame-Based Camera

The framed-based camera is commonly used for object tracking tasks. However, under poor light conditions, the frame camera cannot work properly and provide stable detection for the tracking system. Also, it has motion blur when the object is moving fast. Compared to the frame-based camera, the event camera has a higher temporal resolution and dynamic range. The event data is transmitted immediately when the brightness of each pixel changes. The higher temporal resolution helps the event camera avoid the motion blur and provide more precise temporal information. Moreover, the higher dynamic range allows the event camera to work in extreme light conditions, such as evening, raining, and strong sunshine environments. In all, those advantages make the event cameras an applicable complementary sensor to the frame cameras. In this paper, several optical flow algorithms for the event data are tested. Then, a clustering algorithm is proposed to generate the detection and hybrid it with the detection from frame images. At last, the PMBM filter is used to realize object tracking and test it on our self-recorded dataset with the DAVIS346 and several publicly available datasets.