AAFL: automatic association feature learning for gene signature identification of cancer subtypes in single-cell RNA-seq data

Single-cell RNA-sequencing (scRNA-seq) technologies have enabled the study of human cancers in individual cells, which explores the cellular heterogeneity and the genotypic status of tumors. Gene signature identification plays an important role in the precise classification of cancer subtypes. However, most existing gene selection methods only select the same informative genes for each subtype. In this study, we propose a novel gene selection method, automatic association feature learning (AAFL), which automatically identifies different gene signatures for different cell subpopulations (cancer subtypes) at the same time. The proposed AAFL method combines the residual network with the low-rank network, which selects genes that are most associated with the corresponding cell subpopulations. Moreover, the differential expression genes are acquired before gene selection to filter the redundant genes. We apply the proposed feature learning method to the real cancer scRNA-seq data sets (melanoma) to identify cancer subtypes and detect gene signatures of identified cancer subtypes. The experimental results demonstrate that the proposed method can automatically identify different gene signatures for identified cancer subtypes. Gene ontology enrichment analysis shows that the identified gene signatures of different subtypes reveal the key biological processes and pathways. These gene signatures are expected to bring important implications for understanding cellular heterogeneity and the complex ecosystem of tumors.