Multi-label Semantic Decoding from Human Brain Activity

It is meaningful to decode the semantic information from functional magnetic resonance imaging (fMRI) brain signals evoked by natural images. Semantic decoding can be viewed as a classification problem. Since a natural image may contain many semantic information of different objects, the single label classification model is not appropriate to cope with semantic decoding problem, which motivates the multi-label classification model. However, most multi-label models always treat each label equally. Actually, if dataset is associated with a large number of semantic labels, it will be difficult to get an accurate prediction of semantic label when the label appears with a low frequency in this dataset. So we should increase the relative importance degree to the labels that associate with little instances. In order to improve multi-label prediction performance, in this paper, we firstly propose a multinomial label distribution to estimate the importance degree of each associated label for an instance by using conditional probability, and then establish a deep neural network (DNN) based model which contains both multinomial label distribution and label co-occurrence information to realize the multi-label classification of semantic information in fMRI brain signals. Experiments on three fMRI recording datasets demonstrate that our approach performs better than the state-of-the-art methods on semantic information prediction.