Q-Learning for Non-Cooperative Channel Access Game of Cognitive Radio Networks

This paper investigates the channel access problem of cognitive radio networks. In the cognitive radio network, communication channels are assigned to primary users with priority while secondary users are able to detect the spectrum holes and switch among the channels for data transmission opportunities. The channel access problem of this kind of system can be formulated as a non-cooperative game. However, in prior works, the secondary users are usually assumed to be able to switch to any channel instantaneously, which is not possible in reality because the channel switching will incur transmission delays. In this paper, we formulate the channel access problem as a non-cooperative game where each channel can be used by only one user at a time. Moreover, considering the transmission delays, we limit the channel switching distance of the secondary users to a certain scope. In this case, the optimal channel access policy of each secondary user will depend on the long-term behaviors of primary users as well as the actions of other secondary users. For this non-cooperative game, we propose a multiagent Q-learning algorithm which requires neither the prior knowledge of channel dynamics nor the negotiations among players. Simulation examples are provided to demonstrate the effectiveness of the algorithm.