Variational Skill Embeddings for Meta Reinforcement Learning

Meta reinforcement learning (meta-RL) aims to learn useful prior knowledge across tasks which can be generalized to unseen but similar tasks with only a small number of adaptation steps. Traditionally, the gradient-based metal RL was proposed to use the gradients to learn the parameters of an adaptive policy from different tasks which likely lacked sample efficiency. Recently, the context-based meta-RL improved the efficiency by learning the embeddings of the trajectories based on context representation. The learned policy can be adapted to new tasks, but the performance is bounded due to a simple context encoder. To deal with this insufficiency, this paper presents a novel regularized meta-RL where the generalization of policy is enhanced through a context-based meta-RL where the conditional variational autoencoder consisting of a context-skill encoder and a soft-actor-critic decoder is implemented. The proposed method pursues the model regularization by discovering the shared skill patterns across tasks in implementation of context-based meta-RL. The experiments on a number of benchmark tasks show the merit of variational skill embeddings for regularized meta-RL.