Adaptive state space formation in reinforcement learning

Difficulties are encountered during the application of the reinforcement learning method to real-world problems. One of them is the formation of a discrete state space from a continuous input signal. In the absence of a priori knowledge about input space, a straightforward approach to this problem is to separate the input space into a grid, and to use lookup tables. But this method suffers from the dimensionality problem. Some studies use continuous function approximaters such as neural networks instead of lookup tables. However, when a global basis function such as sigmoid function is used, convergence can't be guaranteed. For this problem we propose a method in which local basis functions are assigned depending on the task requirement. In the initial state of the learning, only one basis function is presented over the entire space. The basis function is divided by the statistical property of locally weighted temporal difference error. We applied the method to an autonomous robot collision avoidance problem, and evaluated the validity of the algorithm.