Exploiting Additive Structure in Factored MDPs for Reinforcement Learning

SDYNA is a framework able to address large, discrete and stochastic reinforcement learning problems. It incrementally learns a FMDP representing the problem to solve while using FMDP planning techniques to build all efficient policy. SPITI, an instantiation of SDYNA, uses a planning method based oil dynamic programming which cannot exploit the additive structure of a FMDP. In this paper, we present two new instantiations of SDYNA, namely ULP and UNATLP, using a linear programming based planning method that can exploit the additive structure. of a FMDP and address problems out of reach of SPITI.