Nowadays, defect detection in industrial field has been rapidly developed and applied, but it also faces some problems. First, it is difficult and expensive to collect defect images, leading to the problem of small samples. Networks lack the ability of generalization. Second, current neural networks are limited to specific industrial scenarios for learning and training and hard to be applied on a new domain, thus lack the cross-domain migration capability. Based on the above problems, we propose the concept of cross-domain data joint learning and a two-stage Cross-Domain Defect Detection Network(C3DN) based on segmentation network and classification network, trying to mine the hidden value in crossdomain data. The segmentation part can effectively extract defects from different material textures and locate them while the classification part can focus on the defects with attentional mechanism and tell whether there are defects. In order to verify the feasibility of cross-domain data joint learning, we organized and re-annotated the public datasets of various industrial fields to form a new cross-domain dataset. C3DN had a strong performance in both validation set and test set, showing its good generalization ability. Through the crossdomain defect detection confusion matrix, the excellent performance of C3DN in different industrial fields was compared and verified, showing its good cross-domain migration ability.