Sparse Linear Contextual Bandits via Relevance Vector Machines

This paper describes a linear multi-armed bandit algorithm that exploits sparsity in the underlying unknown weight vector controlling rewards. In linear multi-armed bandits, a user chooses a sequence of (slot machine) "arms" to pull, and each arm pull results in the user receiving a stochastic reward with mean equal to the inner product between a known feature vector associated with the arm and an unknown weight vector. While linear bandit algorithms have been widely considered in the literature, relatively little is known about how to exploit sparsity in the weight vector. This paper describes a novel approach that leverages ideas from linear Thompson sampling and relevance vector machines, resulting in a scalable approach that adapts to the unknown sparse support. Theoretical regret bounds highlight the proposed algorithm's performance as a function of the sparsity level, and simulations illustrate the advantages of the proposed method over several competing approaches.