In the mainstream convolutional neural network structure framework, a single-label classification network uses a backbone network as a feature extraction network to map the image to a global feature vector, and then the global feature vector is input to the classifier for classification. In the context of multi-label image classification, the classification results of multiple types of labels will be fed back to the global feature vector through a back-propagation algorithm. In this process, the backpropagation information of multi-labels will generate a certain degree of competition. Due to the competition between different object categories, some categories with complex features but small sample numbers are difficult to get correctly classified. This paper proposes an end-to-end convolutional neural network based on a multi-path structure, which not only retains the characteristics of end-to-end and parallelizable operations of the convolutional neural network, but also reduces the feature competition between different types of objects and improves The recognition performance of the network. In addition, the structure has the characteristics of easy expansion. For newly added object categories, the network can well continue the previous learning results, quickly complete the classification of new objects, and can also design specific tributary network structures for specific types of objects.