Objective: The objective is to improve the accuracy and speed of diagnosis of esophageal cancer by endoscopists. Method: In this study, based on the semantic segmentation technology based on the deep convolution neural network, a semantic segmentation method is proposed to assist the intelligent diagnosis of esophageal cancer. On this basis, the M-DeepLab network with better segmentation effect and better accuracy is developed. The network is used to segment the narrowband imaging combined with magnifying endoscopy in 450 patients with esophageal cancer in our hospital, and the segmentation method is compared and evaluated by using the overall accuracy MU, sensitivity, specificity and ROC curve. Results: The image segmentation algorithm proposed in this study can segment the lesion area of esophageal cancer well which is basically in line with the doctor's annotation area. The sensitivity, specificity, overall accuracy MloU and ROC curve of subjects of M-DeepLab network are higher than those of FCN-32s (Fully Convolutional Networks) and DeepLabv3+ network, and the segmentation speed can reach 15 frames per second. Conclusion: From the results, it can be concluded that it is feasible to use M-DeepLab network segmentation algorithm to segment and label the lesion area of esophageal cancer, and the advantages of this method are also very high. It can be used in the artificial intelligence system to assist doctors in the diagnosis of esophageal cancer, improve the diagnosis rate and diagnosis speed.