Towards reliable robot packing system based on deep reinforcement learning

Object packing by a robot has a wide range of applications in the logistics industry. This task requires the variable size items to be picked from piles one by one and then packed into another container immediately without the information about the unpicked items, modeled as an Online 3D Bin Packing Problem (3D-BPP). Due to limited information, it is a challenging problem to obtain an optimal solution for maximizing space utilization. Furthermore, existing studies do not consider practical constraints and assume an ideal perception and robotic packing manipulation. In this paper, we present a robot packing system with high performance and reliability. First, the Online 3D-BPP is formulated as a Markov decision process. A deep reinforcement learning (DRL) approach is proposed to tackle the problem utilizing the observations of the container and the current item. Specifically, a candidate map that indicates the potentially feasible placements based on heuristics is introduced to balance the exploration and exploitation in the considerable discrete action space. Second, we develop a physical robotic system to bridge the DRL agent from simulation to practical application. To make the packing manipulation resilient to uncertainties from the physical system, we design a motion primitive by moving the picked item close to its target placement from a collision-free area within the container. Experiments demonstrate that our method delivers superior performance against the baselines on two datasets, improving space utilization by over 2.7% and 3.8%, respectively, and the performance is not limited by the container size. Moreover, our robotic system can facilitate DRL to perform well in the real world.