Objectives: In order to improve the ability of convolutional neural networks (CNNs) of understanding temporal dynamic information, this paper proposes a dominant layer optimization module. Methods: The new module uses the dominant layer to guide and optimize the update gradient of convolutional layer weights, and assist the difference estimation with the maximum mean difference algorithm of a reproducing Hilbert space. Results: In continuous training, the network can improve the learning ability of temporal dynamic information, and the dynamic information similarity between the features learned by convolutional layer and the input data is also increased. Conclusions: This module enhances the performance of the CNNs model on video human action classification and achieves improvements to the network. © 2021, Editorial Board of Geomatics and Information Science of Wuhan University. All right reserved.