Learning Stationary Correlated Equilibria in Constrained General-Sum Stochastic Games

被引:21
|
作者
Hakami, Vesal [1 ]
Dehghan, Mehdi [1 ]
机构
[1] Amirkabir Univ Technol, Dept Comp Engn & Informat Technol, Tehran 15914, Iran
关键词
Asynchronous stochastic approximation; constrained stochastic game; correlated equilibrium (CE); multiagent systems; no-regret learning; Q-learning; KNOWLEDGE;
D O I
10.1109/TCYB.2015.2453165
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
We study constrained general-sum stochastic games with unknown Markovian dynamics. A distributed constrained no-regret Q-learning scheme (CNRQ) is presented to guarantee convergence to the set of stationary correlated equilibria of the game. Prior art addresses the unconstrained case only, is structured with nested control loops, and has no convergence result. CNRQ is cast as a single-loop three-timescale asynchronous stochastic approximation algorithm with set-valued update increments. A rigorous convergence analysis with differential inclusion arguments is given which draws on recent extensions of the theory of stochastic approximation to the case of asynchronous recursive inclusions with set-valued mean fields. Numerical results are given for the exemplary application of CNRQ to decentralized resource control in heterogeneous wireless networks.
引用
收藏
页码:1640 / 1654
页数:15
相关论文
共 50 条
  • [1] General-sum stochastic games: Verifiability conditions for Nash equilibria
    Prasad, H. L.
    Bhatnagar, S.
    AUTOMATICA, 2012, 48 (11) : 2923 - 2930
  • [2] Two-Timescale Algorithms for Learning Nash Equilibria in General-Sum Stochastic Games
    Prasad, H. L.
    Prashanth, L. A.
    Bhatnagar, Shalabh
    PROCEEDINGS OF THE 2015 INTERNATIONAL CONFERENCE ON AUTONOMOUS AGENTS & MULTIAGENT SYSTEMS (AAMAS'15), 2015, : 1371 - 1379
  • [3] Nash Q-learning for general-sum stochastic games
    Hu, JL
    Wellman, MP
    JOURNAL OF MACHINE LEARNING RESEARCH, 2004, 4 (06) : 1039 - 1069
  • [4] Sample-Efficient Learning of Stackelberg Equilibria in General-Sum Games
    Bai, Yu
    Jin, Chi
    Wang, Huan
    Xiong, Caiming
    ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 34 (NEURIPS 2021), 2021, 34
  • [5] Convergence of Policy Gradient Methods for Nash Equilibria in General-sum Stochastic Games
    Chen, Yan
    Li, Tao
    IFAC PAPERSONLINE, 2023, 56 (02): : 3435 - 3440
  • [6] Near-Optimal No-Regret Learning for Correlated Equilibria in Multi-player General-Sum Games
    Anagnostides, Ioannis
    Daskalakis, Constantinos
    Farina, Gabriele
    Fishelson, Maxwell
    Golowich, Noah
    Sandholm, Tuomas
    PROCEEDINGS OF THE 54TH ANNUAL ACM SIGACT SYMPOSIUM ON THEORY OF COMPUTING (STOC '22), 2022, : 736 - 749
  • [7] Computing Stackelberg Equilibria of Large General-Sum Games
    Blum, Avrim
    Haghtalab, Nika
    Hajiaghayi, MohammadTaghi
    Seddighin, Saeed
    ALGORITHMIC GAME THEORY (SAGT 2019), 2019, 11801 : 168 - 182
  • [8] Decentralized Online Learning in General-Sum Stackelberg Games
    Yu, Yaolong
    Chen, Haipeng
    UNCERTAINTY IN ARTIFICIAL INTELLIGENCE, 2024, 244 : 4056 - 4077
  • [9] Multi-agent Inverse Reinforcement Learning for Certain General-Sum Stochastic Games
    Lin, Xiaomin
    Adams, Stephen C.
    Beling, Peter A.
    JOURNAL OF ARTIFICIAL INTELLIGENCE RESEARCH, 2019, 66 : 473 - 502
  • [10] PAC Reinforcement Learning Algorithm for General-Sum Markov Games
    Zehfroosh, Ashkan
    Tanner, Herbert G.
    IEEE TRANSACTIONS ON AUTOMATIC CONTROL, 2023, 68 (05) : 2821 - 2831