Temporal self-attention-based Conv-LSTM network for multivariate time series prediction

被引:44
|
作者
Fu, En [1 ]
Zhang, Yinong [2 ]
Yang, Fan [3 ]
Wang, Shuying [2 ]
机构
[1] Beijing Union Univ, Beijing Key Lab Informat Serv Engn, Beijing 100101, Peoples R China
[2] Beijing Union Univ, Coll Urban Rail Transit & Logist, Beijing 100101, Peoples R China
[3] Tsinghua Univ, Dept Automat, Beijing 100084, Peoples R China
关键词
Self-attention mechanism; Long short-term memory; Multivariate time series; Prediction;
D O I
10.1016/j.neucom.2022.06.014
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Time series play an important role in many fields, such as industrial control, automated monitoring, and weather forecasting. Because there is often more than one variable in reality problems and they are related to each other, the multivariable time series (MTS) introduced. Using historical observations to accurately predict MTS is still very challenging. Therefore, a new time series prediction model proposed based on the temporal self-attention mechanism, convolutional neural network and long short-term memory (Conv-LSTM). When the standard attention mechanism for time series is combined with recurrent neural network (RNN), it heavily depends on the hidden state of the RNN. Particularly in the first time step, the initial hidden state (typically 0) must be artificially introduced to calculate the attention weight of that step, which results in additional noise in the calculation of the attention weight. To address this problem and increase the flexibility of the attention layer, a new self-attention mechanism designed to extract the temporal dependence of the MTS, which called temporal self-attention. In this attention mechanism, long short-term memory (LSTM) adopted as a sequence encoder to calculate the query, key, and value to obtain a more complete temporal dependence than standard self-attention. Because of flexibility of this structure, the DA-Conv-LSTM model was improved, in which a SOTA attention based method used for MTS prediction. Our improved model compared with six baseline models on multiple datasets (SML2010 and NASDAQ100), and applied to satellite state prediction (our private dataset). The effectiveness of our temporal self-attention was demonstrated by experiments. And the best shortterm prediction performance was achieved by our improved model.(c) 2022 Elsevier B.V. All rights reserved.
引用
收藏
页码:162 / 173
页数:12
相关论文
共 50 条
  • [31] Multiscale global and local self-attention-based network for remaining useful life prediction
    Zhang, Zhizheng
    Song, Wen
    Li, Qiqiang
    Gao, Hui
    [J]. MEASUREMENT SCIENCE AND TECHNOLOGY, 2023, 34 (12)
  • [32] A Multivariate Time Series Prediction Schema based on Multi-attention in recurrent neural network
    Yin, Xiang
    Han, Yanni
    Sun, Hongyu
    Xu, Zhen
    Yu, Haibo
    Duan, Xiaoyu
    [J]. 2020 IEEE SYMPOSIUM ON COMPUTERS AND COMMUNICATIONS (ISCC), 2020, : 717 - 723
  • [33] Prediction of heavy metal content in multivariate chaotic time series based on LSTM
    Wang, Shengwei
    Lou, Tianlong
    Zhang, Chang
    Hao, Ji
    Zhan, Yulin
    Ping, Li
    [J]. DESALINATION AND WATER TREATMENT, 2020, 197 : 249 - 260
  • [34] Multi-Attention Generative Adversarial Network for Multivariate Time Series Prediction
    Yin, Xiang
    Han, Yanni
    Sun, Hongyu
    Xu, Zhen
    Yu, Haibo
    Duan, Xiaoyu
    [J]. IEEE ACCESS, 2021, 9 : 57351 - 57363
  • [35] Multivariate time series classification based on spatial-temporal attention dynamic graph neural network
    Qian, Lipeng
    Zuo, Qiong
    Liu, Haiguang
    Zhu, Hong
    [J]. Applied Intelligence, 2025, 55 (02)
  • [36] Temporal pattern attention for multivariate time series forecasting
    Shun-Yao Shih
    Fan-Keng Sun
    Hung-yi Lee
    [J]. Machine Learning, 2019, 108 : 1421 - 1441
  • [37] Temporal pattern attention for multivariate time series forecasting
    Shih, Shun-Yao
    Sun, Fan-Keng
    Lee, Hung-yi
    [J]. MACHINE LEARNING, 2019, 108 (8-9) : 1421 - 1441
  • [38] DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting
    Huang, Siteng
    Wang, Donglin
    Wu, Xuehan
    Tang, Ao
    [J]. PROCEEDINGS OF THE 28TH ACM INTERNATIONAL CONFERENCE ON INFORMATION & KNOWLEDGE MANAGEMENT (CIKM '19), 2019, : 2129 - 2132
  • [39] A novel hybrid deep learning model with ARIMA Conv-LSTM networks and shuffle attention layer for short-term traffic flow prediction
    Sattarzadeh, Ali Reza
    Kutadinata, Ronny J.
    Pathirana, Pubudu N.
    Huynh, Van Thanh
    [J]. TRANSPORTMETRICA A-TRANSPORT SCIENCE, 2023,
  • [40] GAT-DNS: DNS Multivariate Time Series Prediction Model Based on Graph Attention Network
    Lu, Xiaofeng
    Zhang, Xiaoyu
    Lio, Pietro
    [J]. COMPANION OF THE WORLD WIDE WEB CONFERENCE, WWW 2023, 2023, : 127 - 131