On Convergence Rate of MRetrace

被引:0
|
作者
Chen, Xingguo [1 ]
Qin, Wangrong [1 ]
Gong, Yu [1 ]
Yang, Shangdong [1 ]
Wang, Wenhao [2 ,3 ]
机构
[1] Nanjing Univ Posts & Telecommun, Jiangsu Key Lab Big Data Secur & Intelligent Proc, Nanjing 210023, Peoples R China
[2] Natl Univ Def Technol, Coll Elect Engn, Changsha 410073, Peoples R China
[3] Natl Univ Def Technol, Sci & Technol Informat Syst Engn Lab, Changsha 410073, Peoples R China
关键词
finite sample analysis; off-policy learning; minimum eigenvalues; MRetrace;
D O I
10.3390/math12182930
中图分类号
O1 [数学];
学科分类号
0701 ; 070101 ;
摘要
Off-policy is a key setting for reinforcement learning algorithms. In recent years, the stability of off-policy learning for value-based reinforcement learning has been guaranteed even when combined with linear function approximation and bootstrapping. Convergence rate analysis is currently a hot topic. However, the convergence rates of learning algorithms vary, and analyzing the reasons behind this remains an open problem. In this paper, we propose an essentially simplified version of a convergence rate to generate general off-policy temporal difference learning algorithms. We emphasize that the primary determinant influencing convergence rate is the minimum eigenvalue of the key matrix. Furthermore, we conduct a comparative analysis of the influencing factor across various off-policy learning algorithms in diverse numerical scenarios. The experimental findings validate the proposed determinant, which serves as a benchmark for the design of more efficient learning algorithms.
引用
收藏
页数:19
相关论文
共 50 条
  • [31] On the Rate of Convergence of Fictitious Play
    Brandt, Felix
    Fischer, Felix
    Harrenstein, Paul
    THEORY OF COMPUTING SYSTEMS, 2013, 53 (01) : 41 - 52
  • [32] On Rate of Convergence for Universality Limits
    Bessonov, Roman
    INTEGRAL EQUATIONS AND OPERATOR THEORY, 2024, 96 (01)
  • [33] On the rate of strong convergence for convolutions
    Yu. Davydov
    Journal of Mathematical Sciences, 1997, 83 (3) : 393 - 396
  • [34] Exponential convergence rate in entropy
    Chen M.-F.
    Frontiers of Mathematics in China, 2007, 2 (3) : 329 - 358
  • [35] RATE OF CONVERGENCE OF RECURSIVE ESTIMATORS
    GERENCSER, L
    SIAM JOURNAL ON CONTROL AND OPTIMIZATION, 1992, 30 (05) : 1200 - 1227
  • [36] Rate of Convergence for an Integral Solution
    Chen, Hongwei
    AMERICAN MATHEMATICAL MONTHLY, 2018, 125 (07): : 667 - 668
  • [37] Rate of Convergence for Consensus with Delays
    Bliman, Pierre-Alexander
    Nedic, Angelia
    Ozdaglar, Asuman
    47TH IEEE CONFERENCE ON DECISION AND CONTROL, 2008 (CDC 2008), 2008, : 4849 - 4854
  • [38] On the rate of convergence of the ECME algorithm
    Stat Probab Lett, 1 (81):
  • [39] On the Rate of Convergence of Fictitious Play
    Brandt, Felix
    Fischer, Felix
    Harrenstein, Paul
    ALGORITHMIC GAME THEORY, 2010, 6386 : 102 - +
  • [40] RATE OF CONVERGENCE TO NORMAL DISTRIBUTION
    IBRAGIMOV, IA
    DOKLADY AKADEMII NAUK SSSR, 1965, 161 (06): : 1267 - +