Combine Visual Features and Scene Semantics for Image Captioning

Most of the existing image captioning methods only use the visual information of the image to guide the generation of the captions, lacking the guidance of effective scene semantic information. In addition, the current visual attention mechanism cannot adjust the focus intensity on the image effectively. In order to solve these problems, this paper firstly proposes an improved visual attention model, which introduces a focus intensity coefficient so as to adjust attention intensity automatically. Specifically, the focus intensity coefficient of the attention mechanism is a learnable scaling factor. It can be calculated by the image information and the context information of the model at each time step of the language model decoding procedure. When using the attention mechanism to calculate the attention weight distribution on the image, the "soft" or "hard" intensity of attention mechanism can be adjusted automatically by adaptively scaling the input value of softmax function through the focus intensity coefficient. Then the concentration and dispersion of the visual attention can be achieved. Therefore, the proposed attention model can make the extracted image visual information more accurate. Furthermore, we combine unsupervised and supervised learning methods to extract a series of topic words related to the image scene to represent scene semantic information of the image, which is added to the language model to guide the generation of captions. We believe that each image contains several scene topic concepts, and each topic concept can be represented by some topic words. Specifically, we use the latent Direchlet allocation (LDA) model to cluster all the caption texts in the dataset. Then the topic category of the caption text is used to represent the scene category of corresponding image. What is more, we train a multi-layer perceptron (MLP) to classify the image into topic concepts. As a result, each topic category is represented by a series of topic words obtained from clustering. Then the scene semantic information of each image can be represented by these topic words, which are very relevant to the image scene. We add these topic words to the language model so that it can obtain more prior knowledge. Since the topic information of the image scene is obtained through analyzing the captions, it contains some global information of the captions to be generated. Therefore, our model can predict some important words that suitable for image scene. Finally, we use the attention mechanism to determine the visual information of the image and the semantic information of the scene that the model pays attention to at each time step of the decoding procedure, and use the gating mechanism to control the proportion of the input of these two information. Afterwards, both information is combined to guide the model to generate more accurate and scene-specific captions. In the experimental section, we evaluate our model on two standard datasets, i.e. MSCOCO and Flickr30k. The experimental results show that our approach can generate more accurate captions than many state-of-the-art approaches. In addition, compared with the baseline approach, our approach achieves about 3% improvement on overall evaluation metrics. © 2020, Science Press. All right reserved.