Learn multi-step object sorting tasks through deep reinforcement learning

Robotic systems are usually controlled to repetitively perform specific actions for manufacturing tasks. The traditional control methods are domain-dependent and model-dependent with cost of much human efforts. They cannot meet the new requirements of generality and flexibility in many areas such as intelligent manufacturing and customized production. This paper develops a general model-free approach to enable robots to perform multi-step object sorting tasks through deep reinforcement learning. Taking projected heightmap images from different time steps as input without extra high-level image analysis and understanding, critic models are designed to produce a pixel-wise Q value map for each type of action. It is a new trial to apply pixel-wise Q value-based critic networks to solve multi-step sorting tasks that involve many types of actions and complex action constraints. The experimental validations on simulated and realistic object sorting tasks demonstrate the effectiveness of the proposed approach. Qualitative results (videos), code for simulated and realistic experiments, and pre-trained models are available at https://github.com/JiatongBao/DRLSorting