Improved Sample Complexity for Incremental Autonomous Exploration in MDPs

We investigate the exploration of an unknown environment when no reward function is provided. Building on the incremental exploration setting introduced by Lim and Auer [1], we define the objective of learning the set of epsilon-optimal goal-conditioned policies attaining all states that are incrementally reachable within L steps (in expectation) from a reference state s(0). In this paper, we introduce a novel model-based approach that interleaves discovering new states from s(0) and improving the accuracy of a model estimate that is used to compute goal-conditioned policies to reach newly discovered states. The resulting algorithm, DisCo, achieves a sample complexity scaling as (O) over tilde ((LSL+epsilon)-S-5 Gamma(L+epsilon) A epsilon(-2)), where A is the number of actions, SL+epsilon is the number of states that are incrementally reachable from s(0) in L + epsilon steps, and Gamma(L+epsilon) is the branching factor of the dynamics over such states. This improves over the algorithm proposed in [1] in both epsilon and L at the cost of an extra Gamma(L+epsilon) factor, which is small in most environments of interest. Furthermore, DisCo is the first algorithm that can return an epsilon/c(min)-optimal policy for any cost-sensitive shortest-path problem defined on the L-reachable states with minimum cost c(min). Finally, we report preliminary empirical results confirming our theoretical findings.