Fault detection is an important task of seismic interpretation. Recently, deep learning methods have attracted huge attention in the industry as they have substantially improved automation compared to traditional manual or semi-manual based fault detection methods. Currently, most deep -leaning-based fault detection methods are based on semantic segmentation or image classification, but the faults predicted by these methods are noisy. By contrast, the edge detection network resists noise better by learning the discontinuous characteristics of faults in the seismic profile. In order to utilize the anti-noise advantage of edge-detection networks, as well as, improve their performance, in this paper, we use pixel-difference-based convolutional operators to construct a network model named Pixel Difference Networks Pidinet. Compared with the traditional edge detection network, Pidinet combines the traditional edge detection operator with the deep learning network to effectively improve the effect of edge detection. For better predicting faults, we further optimized the original Pidinet by removing some unnecessary branches and convolution layers. The pixel difference-based convolution operator boosts the neural network to learn fault information better than the traditional convolution operators. For fully learning the fault features in the data, a small amount of real seismic data samples is mixed with the synthetic seismic data samples in the training sample set. Tests on both synthetic and real data sets demonstrated that Pidinet improved the Intersection over Union (IoU) by about 10% compared to Holistically Nested Edge Detection (HED) network. Furthermore, to test the transfer-learning ability of the network, only a small amount of data is used to further fine-tune the network. Compared to the classification network, the transfer learning results improved by more than 10% in terms of Fl score and sensitivity. Finally, the publicly available real seismic data are used for the test. The experimental results show that the faults identified by Pidinet are continuous and clear, thus demonstrating the effectiveness of the edge-detection-based deep-learning algorithms for fault detection.