Phosphogypsum backfill mining has been gradually promoted in recent years. However, phosphogypsum suffers from some defects that limit its further application. In this study, based on the response surface method (RSM), an anhydrous phosphogypsum-based backfill material (APBM) activated by all-solid waste was developed. The microscopic morphology and phase composition of the hydration products were investigated by scanning electron microscopy, energy dispersive spectroscopy, and X-ray diffraction. The results showed that: The addition of carbide slag alone reduced the fluidity and bleeding rate of anhydrous phosphogypsum, while the addition of single blast furnace slag was the opposite. The single addition of carbide slag had a significant stimulating effect on the strength of anhydrous phosphogypsum, but it damaged the later strength. The regression model established by the RSM had good accuracy and reliability, and the optimal ratio after multi-objective optimization was 3% carbide slag, 39.8% blast furnace slag, and 34.2% water-cement ratio. The crystal structure of anhydrous phosphogypsum hydration product dihydrate gypsum after adding carbide slag alone was poor in the later stage, and the length-diameter ratio decreased. The C-S(A)-H gel and ettringite formed after incorporating CS and BFS filled the pores between the dihydrate gypsum crystals, which greatly improved the compactness of the APBM. The research results could guide the large-scale utilization of phosphogypsum for backfill mining.