Automatic Classification Framework of Tongue Feature Based on Convolutional Neural Networks

Tongue diagnosis is an important part of the diagnostic process in traditional Chinese medicine (TCM). It primarily relies on the expertise and experience of TCM practitioners in identifying tongue features, which are subjective and unstable. We proposed a tongue feature classification framework based on convolutional neural networks to reduce the differences in diagnoses among TCM practitioners. Initially, we used our self-designed instrument to capture 482 tongue photos and created 11 data sets based on different features. Then, the tongue segmentation task was completed using an upgraded facial landmark detection method and UNET. Finally, we used ResNet34 as the backbone to extract features from the tongue photos and classify them. Experimental results show that our framework has excellent results with an overall accuracy of over 86 percent and is particularly sensitive to the corresponding feature regions, and thus it could assist TCM practitioners in making more accurate diagnoses.