Relative molecule self-attention transformer

被引:2
|
作者
Maziarka, Lukasz [1 ]
Majchrowski, Dawid [2 ]
Danel, Tomasz [1 ]
Gainski, Piotr [1 ,3 ]
Tabor, Jacek [1 ]
Podolak, Igor [1 ]
Morkisz, Pawel [2 ]
Jastrzebski, Stanislaw [4 ]
机构
[1] Jagiellonian Univ, Fac Math & Comp Sci, Lojasiewicza 6, PL-30348 Krakow, Poland
[2] NVIDIA, 2788 San Tomas Expy, Santa Clara, CA 95051 USA
[3] Ardigen, Podole 76, PL-30394 Krakow, Poland
[4] Molecule one, Al Jerozolimskie 96, PL-00807 Warsaw, Poland
关键词
Molecular property prediction; Molecular self-attention; Neural networks pre-training; MACHINE LEARNING-METHODS;
D O I
10.1186/s13321-023-00789-7
中图分类号
O6 [化学];
学科分类号
0703 ;
摘要
The prediction of molecular properties is a crucial aspect in drug discovery that can save a lot of money and time during the drug design process. The use of machine learning methods to predict molecular properties has become increasingly popular in recent years. Despite advancements in the field, several challenges remain that need to be addressed, like finding an optimal pre-training procedure to improve performance on small datasets, which are common in drug discovery. In our paper, we tackle these problems by introducing Relative Molecule Self-Attention Transformer for molecular representation learning. It is a novel architecture that uses relative self-attention and 3D molecular representation to capture the interactions between atoms and bonds that enrich the backbone model with domain-specific inductive biases. Furthermore, our two-step pretraining procedure allows us to tune only a few hyperparameter values to achieve good performance comparable with state-of-the-art models on a wide selection of downstream tasks. A novel graph transformer architecture for molecular property prediction is introduced. The task-agnostic methodology for pre-training this model is presented, improving target task performance with minimal hyperparameter tuning. A rigorous exploration of the design space for the self-attention layer is conducted to identify the optimal architecture.
引用
收藏
页数:14
相关论文
共 50 条
  • [1] Relative molecule self-attention transformer
    Łukasz Maziarka
    Dawid Majchrowski
    Tomasz Danel
    Piotr Gaiński
    Jacek Tabor
    Igor Podolak
    Paweł Morkisz
    Stanisław Jastrzębski
    [J]. Journal of Cheminformatics, 16
  • [2] Universal Graph Transformer Self-Attention Networks
    Dai Quoc Nguyen
    Tu Dinh Nguyen
    Dinh Phung
    [J]. COMPANION PROCEEDINGS OF THE WEB CONFERENCE 2022, WWW 2022 COMPANION, 2022, : 193 - 196
  • [3] SST: self-attention transformer for infrared deconvolution
    Gao, Lei
    Yan, Xiaohong
    Deng, Lizhen
    Xu, Guoxia
    Zhu, Hu
    [J]. INFRARED PHYSICS & TECHNOLOGY, 2024, 140
  • [4] Sparse self-attention transformer for image inpainting
    Huang, Wenli
    Deng, Ye
    Hui, Siqi
    Wu, Yang
    Zhou, Sanping
    Wang, Jinjun
    [J]. PATTERN RECOGNITION, 2024, 145
  • [5] Lite Vision Transformer with Enhanced Self-Attention
    Yang, Chenglin
    Wang, Yilin
    Zhang, Jianming
    Zhang, He
    Wei, Zijun
    Lin, Zhe
    Yuille, Alan
    [J]. 2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2022, : 11988 - 11998
  • [6] Synthesizer: Rethinking Self-Attention for Transformer Models
    Tay, Yi
    Bahri, Dara
    Metzler, Donald
    Juan, Da-Cheng
    Zhao, Zhe
    Zheng, Che
    [J]. INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 139, 2021, 139 : 7192 - 7203
  • [7] Self-Attention Mechanism in GANs for Molecule Generation
    Chinnareddy, Sandeep
    Grandhi, Pranav
    Narayan, Apurva
    [J]. 20TH IEEE INTERNATIONAL CONFERENCE ON MACHINE LEARNING AND APPLICATIONS (ICMLA 2021), 2021, : 57 - 60
  • [8] Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention
    Pan, Xuran
    Ye, Tianzhu
    Xia, Zhuofan
    Song, Shiji
    Huang, Gao
    [J]. 2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR, 2023, : 2082 - 2091
  • [9] Local self-attention in transformer for visual question answering
    Shen, Xiang
    Han, Dezhi
    Guo, Zihan
    Chen, Chongqing
    Hua, Jie
    Luo, Gaofeng
    [J]. APPLIED INTELLIGENCE, 2023, 53 (13) : 16706 - 16723
  • [10] Tree Transformer: Integrating Tree Structures into Self-Attention
    Wang, Yau-Shian
    Lee, Hung-Yi
    Chen, Yun-Nung
    [J]. 2019 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING AND THE 9TH INTERNATIONAL JOINT CONFERENCE ON NATURAL LANGUAGE PROCESSING (EMNLP-IJCNLP 2019): PROCEEDINGS OF THE CONFERENCE, 2019, : 1061 - 1070