Multi-modal hierarchical fusion network for fine-grained paper classification

Because huge amount of scientific papers have been published at an accelerating rate, it is beneficial to do intelligent paper classification, especially fine-grained classification. However, existing natural language processing techniques are mostly coarse-grained. Some characteristics of fine-grained scientific paper classification needs special attention. One is that the number of data may well be quite limited. Number of papers in the lower level sub-fields inevitably becomes less. Meanwhile, emerging sub-fields with new discoveries will have few papers, nevertheless these sub-fields can be important. Furthermore, fine-grained labeling of scientific papers requires high expertise and is time consuming. Another aspect of scientific papers is that they contain multi-modal information. To address the above two issues, we propose a multi-modal hierarchical fusion network (MHFNet) for fine-grained paper classification. We treat paper abstract features, image features, and paper title features as three modalities. The MobileNetV2 model and the ALBERT model are combined in the proposed model to encode multi-modal information. Comparison results with baseline methods on both sufficiently large datasets and number-limited datasets show improvements, even more on number-limited datasets.