A deep attention-based ensemble network for real-time face hallucination

Face hallucination (FH) aims to reconstruct high-resolution faces from low-resolution face inputs, making it significant to other face-related tasks. Different from general super resolution issue, it often requires facial priors other than general extracted features thus leading to fusion of more than one kind of feature. The existing CNN-based FH methods often fuse different features indiscriminately which may introduce noises. Also the latent relations among different features which may be useful are taken into less consideration. To address the above issues, we propose an end-to-end deep ensemble network which aggregates three extraction sub-nets in attention-based manner. In our ensemble strategy, both relations among different features and inter-dependencies among different channels are dug out through the exploitation of spatial attention and channel attention. And for the diversity of extracted features, we aggregate three different sub-nets, which are the basic sub-net for basic features, the auto-encoder sub-net for facial shape priors and the dense residual attention sub-net for fine-grained texture features. Conducted ablation studies and experimental results show that our method achieves effectiveness not only in PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index) metrics but more importantly in clearer details within both key facial areas and whole range. Also results show that our method achieves real-time hallucinating faces by generating one image in 0.0237s.