Conceptual Sentence Embeddings Based on Attention Mechanism

被引：0

作者：

Wang Y.-S. ^{[1
]}

Huang H.-Y. ^{[1
]}

Feng C. ^{[1
]}

Zhou Q. ^{[2
]}

机构：

[1] Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications, School of Computer, Beijing Institute of Technology, Beijing

[2] Baidu Inc., Beijing

来源：

Zidonghua Xuebao/Acta Automatica Sinica | 2020年 / 46卷 / 07期

基金：

中国国家自然科学基金; 国家自然科学基金重点项目;

关键词：

Attention mechanism; Semantic representation; Sentence embedding; Short-text conceptualization; Word embedding;

D O I：

10.16383/j.aas.2018.c170295

中图分类号：

学科分类号：

摘要：

Most sentence embedding models typically represent each sentence only using word surface, which makes these models indiscriminative for ubiquitous homonymy and polysemy. In order to enhance representation capability of sentence, we employ short-text conceptualization algorithm to assign associated concepts for each sentence in the text corpus, and then learn conceptual sentence embedding (CSE). Hence, this semantic representation is more expressive than some widely-used text representation models such as latent topic model, especially for short-text. Moreover, we further extend CSE models by utilizing an attention mechanism that select relevant words within the context to make more efficient prediction. In the experiments, we evaluate the CSE models on three tasks, text classification and information retrieval. The experimental results show that the proposed models outperform typical sentence embed-ding models. Copyright © 2020 Acta Automatica Sinica. All rights reserved.

引用

页码：1390 / 1400

页数：10

共 39 条

[1] Harris Z S., Distributional Structure, Word, 10, pp. 146-162, (1954)
[2] Palangi H, Deng L, Shen Y, Gao J, He X, Chen J, Song X, Ward R., Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24, pp. 694-707, (2016)
[3] Le Q, Mikolov T., Distributed Representations of Sentences and Documents, Proceedings of the 31st International Conference on Machine Learning, pp. 1188-1196, (2014)
[4] Liu Y, Liu Z, Chua T S, Sun M., Topical Word Embeddings, Proceedings of the 29th AAAI Conference on Artificial Intelligence, pp. 2418-2424, (2015)
[5] Mikolov T, Corrado G, Chen K, Dean J., Efficient Estimation of Word Representations in Vector Space, (2015)
[6] Wang Z, Zhao K, Wang H, Dean J., Query Understanding through Knowledge-Based Conceptualization, Proceed- ings of the 24th International Joint Conference on Artificial Intelligence, pp. 3264-3270, (2015)
[7] Bahdanau D, Cho K, Bengio Y., Neural Machine Translation by Jointly Learning to Align and Translate, (2015)
[8] Lin Y, Shen S, Liu Z, Luan H, Sun M., Neural Relation Extraction with Selective Attention over Instances, Proceedings of the 54th Meeting of the Association for Computational Linguistics, pp. 2124-2133, (2016)
[9] Wang Y, Huang H, Feng C, Zhou Q, Gu J, Gao X., CSE: Conceptual Sentence Embeddings based on Attention Model, Proceedings of the 54th Meeting of the Association for Computational Linguistics, pp. 505-515, (2016)
[10] Rayner K., Eye movements in Reading and Information Processing: 20 Years of Research, Psychological Bulletin, 124, pp. 372-422, (1998)

← 1 2 3 4 →