Generalized Policy Iteration-based Reinforcement Learning Algorithm for Optimal Control of Unknown Discrete-time Systems

This paper presents a novel generalized policy iteration-based reinforcement learning (RL) algorithm to deal with infinite-horizon optimal control problems of nonlinear discrete-time systems with completely unknown dynamics. In the present iterative algorithm, two iteration procedures are utilized to obtain the iterative Q-function and the iterative control policy. Furthermore, the iterative Q-function is obtained by the temporal difference learning and the policy gradient method is utilized to directly optimize the iterative control policy. Then, the convergence and optimality analysis of the generalized policy iteration-based RL algorithm are provided. To implement this algorithm, two neural networks, including a critic network and an action network, are used to approximate the iterative Q-function and the iterative control policy. Finally, a numerical simulation example is provided to illustrate the effectiveness of the proposed control method.