Pre-trained Transformer-Based Citation Context-Aware Citation Network Embeddings

Academic papers form citation networks wherein each paper is a node and citation relationships between papers are edges. The embeddings of each paper obtained by projecting the citation network into a vector space are called citation network embeddings. Thus far, only a limited number of studies have focused on incorporating information regarding the intent of one paper to cite another paper. We consider citation context, i.e., the text to cite a paper, as a source of information for citation intent, and propose a new method for generating citation context-aware citation network embeddings. We trained SciBERT with our proposed masked paper prediction task in which the model predicts the cited paper from the citing paper and the citation context. In addition, we propose a new loss function that considers not only the citation context but also the neighboring nodes in the citation network. We conducted experiments involving citation-recommendation and paper-classification tasks which we formulated on two existing datasets: FullTextPeerRead and AASC. For both tasks, the proposed method outperformed hyperdoc2vec, an existing method for citation context-aware citation network embedding; further, it achieved a comparable performance to a state-of-the-art citation network embedding that do not utilize any citation context for paper classification.