Distributed Frequency Allocation Based on MARL in Dynamic Environment of Cognitive Radio Networks: A Stochastic Game Model

We investigate distributed frequency allocation problem in dynamic environment of cognitive radio (CR) networks with a stochastic game (SG) model. As traditional multi-agent reinforcement learning (MARL) algorithms are not suitable for this problem, a new MARL algorithm, maximizing the average Q function algorithm (MAQ), is proposed in this article. With MAQ, distributed agents can realize indirect coordination without exchanging their rewards or Q values, because they consider other agents' strategies during decision-making. A new equilibrium state, Q function equilibrium (QE), is defined and the agents' strategies achieve QE definitely through MAQ learning. Specifically, the defined QE is proved to be Nash equilibrium when the changing state of primary network is stationary stochastic procedure. Throughput performance of MAQ is close to that of centric learning method, but it needs less or doesn't need intercommunications.