Convergence of Recurrent Neuro-Fuzzy Value-Gradient Learning With and Without an Actor

In recent years, a gradient of the n-step temporaldifference [TD(lambda)] learning has been developed to present an advanced adaptive dynamic programming (ADP) algorithm, called value-gradient learning [VGLl]. In this paper, we improve the VGLl architecture, which is called the "single adaptive actor network [SNVGLl]" because it has only a single approximator function network (critic) instead of dual networks (critic and actor) as in VGLl. Therefore, SNVGLl has lower computational requirements when compared to VGLl. Moreover, in this paper, a recurrent hybrid neuro-fuzzy (RNF) and a first-order Takagi-Sugeno RNF (TSRNF) are derived and implemented to build the critic and actor networks. Furthermore, we develop the novel study of the theoretical convergence proofs for both VGLl and SNVGLl under certain conditions. In this paper, mobile robot simulation model (model based) is used to solve the optimal control problem for affine nonlinear discrete-time systems. Mobile robot is exposed various noise levels to verify the performance and to validate the theoretical analysis.