Semantic segmentation-based observation pose estimation method for tomato harvesting robots

Accurate identification and localization of peduncle cutting points are crucial for the automated harvesting of tomatoes. Due to the slender nature of tomato peduncles, occlusions from surrounding fruits, stems, and other obstacles often occur, which can adversely affect the accuracy of harvesting point detection. An optimal observation viewpoint of the tomato clusters can significantly enhance the visibility of peduncles within the camera frame. This study presents a pose estimation method for tomato cluster observation based on semantic segmentation, aimed at improving peduncle recognition accuracy from the end-effector camera's perspective. A lightweight semantic segmentation network, Dual-Resolution Network with Convolutional Attention (DRCANet), is developed to efficiently identify tomatoes and stems in harvesting scenes. The DRCANet adopts a dual-branch structure that incorporates the Convolutional Attention (CA) Block in the low-resolution semantic branch to enable more efficient semantic feature extraction. Further optimization of model performance is achieved by integrating a Multi-Scale Convolution with Channel Excitation Module (MSCEM), the adaptive-weighted-fusion module (AWF), and shallow feature fusion. The proposed DRCANet predicts masks for both tomatoes and stems in the images. By combining these predicted masks with depth information, the spatial point cloud data of tomatoes and stems are extracted. The spatial relationship between each tomato cluster and its corresponding stem is then analyzed, leading to the final observation pose estimation for each tomato cluster. Experimental results demonstrate that the proposed DRCANet achieves mIoU and mPA values of 82.83 % and 91.37 %, respectively, with an average inference time of 11.42 ms. The proposed observation pose estimation method achieves an accuracy of 77.84 % with an average processing time of 68.25 ms. This study validates the effectiveness of optimizing the observation perspective in improving the recognition accuracy of tomato peduncle picking points, offering a novel approach to enhancing the harvesting success rate of tomato harvesting robots.