Constructing and Evaluating Options in Reinforcement Learning

In this paper, we propose a new subgoal based method for automatic construction of useful options. In our proposed method, subgoals are considered as border states of communities of the transition graph created after some initial agent interactions with the environment. We present a new community detection algorithm to provide an appropriate partitioning of the transition graph. Macro-actions are constructed for taking the agent from one community to other communities. In addition, we attempt to capture intuitions about features of useful macro-actions. There is a lack of a generic evaluation mechanism for evaluating each macro-action in previous research. We will propose a method for evaluating each macro-action separately. Inappropriate macro-actions are identified with this method and discarded from agent choices. Experimental results show a significant improvement in results after pruning macro-actions.