An effective picking point localization method for multi-posture lotus pods based on three-view depth vision observation

Different lotus pod individuals have different growth postures, and conventional single-view observation makes it difficult to ensure the visibility of the stalk's picking segment of all lotus pods, thus affecting the efficiency and quality of automatic picking. To solve this problem, this study proposes a picking point localization method for multi-posture lotus pods based on three-view depth vision observation. It includes three steps: first, the RGB-D (red-green-blue-depth) images of lotus pods are collected from three viewing directions based on the designed three-view depth vision observation scheme. Second, instance segmentation on the lotus pods and stalks in the collected images is performed to obtain mask data. Third, the final lotus pod picking point coordinates and picking vector data are obtained using the proposed lotus pod picking information calculation algorithm. The algorithm comprises the two-dimensional image feature point calculation, single-view picking information calculation, and three-view picking information data synthesis. The three-view observation test of lotus pods in a planting environment shows that the proposed three-view observation scheme can meet the picking segment observation needs of lotus pods in any posture. In addition, lotus pod localization tests were conducted based on a specifically designed localization test bench. The proposed localization method achieved a localization success rate of 98.00 % and 90.86 % for single and multiple lotus pods, respectively. The results verify the feasibility of the proposed localization method for calculating the picking information of arbitrary lotus pods, which can provide essential support for the practical development of automatic lotus pod harvesting technology.