Automatic image caption generation using deep learning and multimodal attention

We present an improved image caption generation model that incorporating multimodal attention mechanism. We use ResNet-101 to extract image features while incorporating channel attention mechanism and spatial attention mechanism. We use Faster R-CNN for object detection and use a multi-head attention structure consisting of spatial attention and self-attention. This allows our algorithm to improve the model's capability to learn and use the internal grammatical features of natural sentences. Moreover, we use GPU parallel computing to accelerate the entire model training. We apply our model and algorithm to early education scenarios: show and tell for kids. We compare our algorithm with the state-of-the-art deep learning algorithms. Our experimental results show that our model improves the captioning accuracy in terms of standard automatic evaluation metrics.