Dictionary Learning-Based Reinforcement Learning with Non-convex Sparsity Regularizer

Spare representations can help improve value prediction and control performances in Reinforcement Learning (RL), by capturing most essential features from states and ignoring unnecessary ones to avoid interference. However, existing sparse coding-based RL methods for control problems are optimized in the neural network methodology, which can not guarantee convergence. To this end, we propose a dictionary learning-based RL with the non-convex sparsity regularizer for RL control. To avoid the black-box optimization with the SGD, we employ the dictionary learning model in RL control, guaranteeing efficient convergence in control experiments. To obtain accurate representations in RL, we employ the non-convex l(p) norm (0 < p < 1) beyond the convex l(1) norm as the sparsity regularizer in dictionary learning-based RL, for capturing more essential features from states. To obtain solutions efficiently, we employ the proximal splitting method to update the multivariate optimization problem. Hence, the non-convex sparsity regularized dictionary learning-based RL is developed and validated in different benchmark RL environments. The proposed algorithm can obtain the best control performances among compared sparse coding-based RL methods with around 10% increases in reward. Moreover, the proposed method can obtain higher sparsity in representations in different environments.