Image moments have attracted extensive attention from researchers because of their good global feature description ability and geometric invariance. In recent years, with the wide dissemination of multi-channel images such as color images, stereoscopic images, and multi-view images, research on the corresponding hypercomplex moments has rapidly developed. Hypercomplex continuous orthogonal moments (HCOMs) are an important branch of hypercomplex moments, which play an important role in solving the problem of geometric invariance of various types of multi-channel images, but the complex computation process leads to their slower computation speed, which in turn affects their application in more scenarios. In this paper, we analyze in detail the factors affecting the computation and reconstruction speed of HCOMs, and propose a fast algorithm for HCOMs, named fast hypercomplex continuous orthogonal moments (FHCOMs). The algorithm optimizes the computation of radial basis function and angular Fourier factor, avoids repeated computations between multiple channels, and reduces the number of computed and reconstructed moments by exploiting the symmetric property of moments. Theoretical analysis and various experimental results show that FHCOMs reduce the computation and reconstruction time by at least 50% and reduce the time consumption for zero- watermarking and watermarking of multi-channel images by nearly 50% compared to HCOMs. Therefore, the proposed FHCOMs have a significant acceleration effect.