In order to meet the high requirements of the magnetic tile production industry for surface quality inspection and realize the automatic segmentation and recognition of magnetic tile defects, a defect segmentation and classification network based on convolutional neural networks is proposed. The network is based on the U-net architecture. The deep features of defects are extracted through the U-net encoding part, and the deep features are used for defect classification, and then the segmented defect areas are output through the decoding part. In order to solve the problem that the proportion of the foreground area of some defects is too small, which makes the network difficult to converge, the continuous optimization of the network is ensured by adding the difference coefficient loss. Then, adding multiple layers of loss and performing online data enhancement in the training phase further improves the segmentation accuracy and classification accuracy. Experimental results show that with the addition of auxiliary loss function and data enhancement, the segmentation network can segment 94.5% of the marked defect areas, and the accuracy of defect classification can reach 98.9%, which meets the high precision requirements of the industry. This method can accurately and effectively segment and identify the surface defects of the magnetic tile, which provides a new idea for the automatic industry of magnetic tile surface quality inspection.