Automated State Feature Learning for Actor-Critic Reinforcement Learning through NEAT

Actor-Critic (AC) algorithms are important approaches to solving sophisticated reinforcement learning problems. However, the learning performance of these algorithms rely heavily on good state features that are often designed manually. To address this issue, we propose to adopt an evolutionary approach based on NeuroEvolution of Augmenting Topology (NEAT) to automatically evolve neural networks that directly transform the raw environmental inputs into state features. Following this idea, we have successfully developed a new algorithm called NEAT+AC which combines Regular-gradient Actor-Critic (RAC) with NEAT. It can simultaneously learn suitable state features as well as good policies that are expected to significantly improve the reinforcement learning performance. Preliminary experiments on two benchmark problems confirm that our new algorithm can clearly outperform the baseline algorithm, i.e., NEAT.