Deep learning-based multi-task prediction of response to neoadjuvant chemotherapy using multiscale whole slide images in breast cancer:A multicenter study

Objective: Early predicting response before neoadjuvant chemotherapy(NAC) is crucial for personalized treatment plans for locally advanced breast cancer patients. We aim to develop a multi-task model using multiscale whole slide images(WSIs) features to predict the response to breast cancer NAC more finely.Methods: This work collected 1,670 whole slide images for training and validation sets, internal testing sets,external testing sets, and prospective testing sets of the weakly-supervised deep learning-based multi-task model(DLMM) in predicting treatment response and pCR to NAC. Our approach models two-by-two feature interactions across scales by employing concatenate fusion of single-scale feature representations, and controls the expressiveness of each representation via a gating-based attention mechanism.Results: In the retrospective analysis, DLMM exhibited excellent predictive performance for the prediction of treatment response, with area under the receiver operating characteristic curves(AUCs) of 0.869 [95% confidence interval(95% CI): 0.806-0.933] in the internal testing set and 0.841(95% CI: 0.814-0.867) in the external testing sets. For the pCR prediction task, DLMM reached AUCs of 0.865(95% CI: 0.763-0.964) in the internal testing and 0.821(95% CI: 0.763-0.878) in the pooled external testing set. In the prospective testing study, DLMM also demonstrated favorable predictive performance, with AUCs of 0.829(95% CI: 0.754-0.903) and 0.821(95% CI:0.692-0.949) in treatment response and pCR prediction, respectively. DLMM significantly outperformed the baseline models in all testing sets(P<0.05). Heatmaps were employed to interpret the decision-making basis of the model. Furthermore, it was discovered that high DLMM scores were associated with immune-related pathways and cells in the microenvironment during biological basis exploration.Conclusions: The DLMM represents a valuable tool that aids clinicians in selecting personalized treatment strategies for breast cancer patients.