Cardiovascular diseases, including all types of arrhythmias, are the leading cause of death. Deep learning (DL)based electrocardiography (ECG) diagnosis systems have attracted considerable attention in recent years. However, training DL-based models requires much high-quality labeled data, and labeling ECG records is timeconsuming and expensive. In this paper, a high-precision deep network obtained via low-cost annotation, that is, a self-supervised residual convolutional neural network (SSRCNN), was designed. First, a convolutional neural network and residual blocks were designed to construct a deep feature extractor to automatically obtain the feature expression of ECG signals. Next, we developed time- and lead-dimension-based data augmentation methods and designed a pretraining framework based on unlabeled datasets such that the feature extractor could update the weights in the unlabeled samples. Furthermore, through interactions with clinicians, we used a few labeled data to fine-tune the pretrained model and feature classifier to automatically extract deep features and perform effective classification. Finally, we used different open-source datasets to validate the superiority of the SSRCNN. With the support of unlabeled datasets, SSRCNN can effectively reduce clinicians' workload. Compared with existing methods, the SSRCNN achieves a better diagnostic performance in terms of average accuracy (99.0 %) and average F1-macro (86.83 %) using approximately 25% of labeled data. Therefore, the SSRCNN has a potential for practical applications in clinical settings.