STCGAN: a novel cycle-consistent generative adversarial network for spatial transcriptomics cellular deconvolution

Motivation Spatial transcriptomics (ST) technologies have revolutionized our ability to map gene expression patterns within native tissue context, providing unprecedented insights into tissue architecture and cellular heterogeneity. However, accurately deconvolving cell-type compositions from ST spots remains challenging due to the sparse and averaged nature of ST data, which is essential for accurately depicting tissue architecture. While numerous computational methods have been developed for cell-type deconvolution and spatial distribution reconstruction, most fail to capture tissue complexity at the single-cell level, thereby limiting their applicability in practical scenarios.Results To this end, we propose a novel cycle-consistent generative adversarial network named STCGAN for cellular deconvolution in spatial transcriptomic. STCGAN first employs a cycle-consistent generative adversarial network (CGAN) to pre-train on ST data, ensuring that both the mapping from ST data to latent space and its reverse mapping are consistent, capturing complex spatial gene expression patterns and learning robust latent representations. Based on the learned representation, STCGAN then optimizes a trainable cell-to-spot mapping matrix to integrate scRNA-seq data with ST data, accurately estimating cellular composition within each capture spot and effectively reconstructing the spatial distribution of cells across the tissue. To further enhance deconvolution accuracy, we incorporate spatial-aware regularization that ensures accurate cellular distribution reconstruction within the spatial context. Benchmarking against seven state-of-the-art methods on five simulated and real datasets from various tissues, STCGAN consistently delivers superior cell-type deconvolution performance.Availability The code of STCGAN can be downloaded from https://github.com/cs-wangbo/STCGAN and all the mentioned datasets are available on Zenodo at https://zenodo.org/doi/10.5281/zenodo.10799113.