End-to-End Learning Deep CRF Models for Multi-Object Tracking Deep CRF Models

By bundling multiple complex sub-problems into a unified framework, end-to-end deep learning frameworks reduce the need for hand engineering or tuning of parameters for each component, and optimize different modules jointly to ensure the generalization of the whole deep architecture. Despite tremendous success in numerous computer vision tasks, end-to-end learnings for multi-object tracking (MOT), especially for the assignment problem in data association, have been surprisingly less investigated mainly due to the lack of available training data. Furthermore, it is challenging to discriminate target objects under mutual occlusions or to reduce identity switches in crowded scenes. To tackle these challenges, this paper proposes learning deep conditional random field (CRF) networks, aiming to model the assignment costs as unary potentials and the long-term dependencies among detection results as pairwise potentials. Specifically, we use a bidirectional long short-term memory (LSTM) network to encode the long-term dependencies. We pose the CRF inference as a recurrent neural network learning process using the standard gradient descent algorithm, where unary and pairwise potentials are jointly optimized in an end-to-end manner. Extensive experiments are conducted on the challenging MOT datasets including MOT15, MOT16 and MOT17, and the results show that the proposed algorithm performs favorably against the state-of-the-art methods.