Hierarchical Nash-Q Learning in Continuous Games

Multi-agent Reinforcement Learning (RL) algorithms usually work on repeated extended, or stochastic games. Generally RL is developed for discrete systems both in terms of states and actions. In this paper, a hierarchical method to learn equilibrium strategy in continuous games is developed. Hierarchy is used to break the continuous domain of strategies into discrete sets of hierarchical strategies. The algorithm is proved to converge to Nash-Equilibrium in a specific class of games with dominant strategies. Then, it is applied to some other games and the convergence in shown. This approach is common in RL algorithms that they are applied to problem where no proof of convergence exits.