Contextual Combinatorial Bandits in Real-Time Strategy Games

The contextual bandit problem is a richer framework than stochastic bandits that has many applications since it allows the learner has access to additional information (the "context"). This additional information can help predict the expected utility of the different arms in many cases. Moreover, combinatorial bandits are a class of bandit problem where the space of possible arms to choose from has a combinatorial structure. In this paper, we investigate the bandit problem where we have both contextual information and there is a combinatorial arm structure, which we call contextual combinatorial bandits (CCMABs). We apply contextual combinatorial bandits to real-time strategy (RTS) games, and study different algorithms to solve CCMABs with different trade-offs of computational efficiency and learning biases. Specifically, we focus on the problem of determining map-specific game playing policies, and formulate it as a CCMABs.