Self-Supervised Learning for 3-D Point Clouds Based on a Masked Linear Autoencoder

Motivated by the success of a masked autoencoder in 3-D point-cloud-based learning, this study proposes an innovative framework for self-supervised learning (SSL) on 3-D point clouds with linear complexity. In the proposed framework, every input point cloud is divided into multiple point patches, which are randomly masked at different ratios. Then, unmasked point patches are then fed to an improved transformer model, which uses an advanced linear self-attention mechanism autoencoder to learn high-level features. The pretraining objective is to recover the masked patches under the guidance of the unmasked point patches' features obtained by the designed transformer. Furthermore, a linear self-attention mechanism is designed to use three projection matrices to decompose the original scaled dot-product attention into smaller parts, using the properties of low-rank and linear decomposition to reduce the time complexity from quadratic to linear. The results of extensive experiments demonstrate that the proposed pretrained model can achieve high accuracy of 93.6% and 84.77% on the ModelNet40 and ScanObjectNN datasets, respectively, at a masking ratio of 40%. In addition, the results show that the proposed method, which uses a linear self-attention mechanism, can enhance the computational efficiency by significantly reducing the inference time and minimizing the storage memory requirements for query, key, and value (Q, K, and V) matrices compared with the existing methods. Finally, the results indicate that the proposed method can achieve state-of-the-art performance on the classification ModelNet40 dataset.