Hybrid Dictionary Learning and Matching for Video-based Face Verification

We propose a hybrid dictionary learning and matching approach using deep features for unconstrained video-based face verification. Popular off-the-shelf image-based deep neural networks often fail to effectively exploit multiple frames for video-based verification. Unlike recurrent neural network-based approaches which require an external large-scale annotated data for training, the proposed unsupervised approach can effectively model both structural and temporal information of face features in target videos using structural and dynamical dictionaries, respectively. We propose an iterative optimization procedure to learn the dynamical dictionaries from videos. Using the learned dictionaries, we model video-to-video similarity as subspace-to-subspace similarity which is not only more robust but also utilizes the information in multiple frames better than the widely used reconstruction error-based measures, where the subspaces are spanned by the learned dictionaries. Experiments on challenging video-based face verification datasets, including Multiple Biometric Grand Challenge (MBGC), Face and Ocular Challenge Series (FOCS) and IARPA JANUS Benchmark A (IJB-A) datasets, demonstrate that the proposed method can effectively learn robust and discriminative representations for videos and improve the face verification performance.