Background: Soy allergy is an important nutritional and health issue that needs to be addressed. Lactic acid bacteria (LAB) fermentation is an effective approach to reduce soy protein allergy. Polysaccharides are commonly used in LAB-fermented products to enhance their textural properties. This study proposes a new strategy for developing hypoallergenic soy protein products. Methods: We prepared a soy protein isolate (SPI) through fermentation with LAB (FSPI) and with five types of polysaccharides supplementation, namely polydextrose (PDX), inulin (IN), long-chain inulin (LCIN), soluble soy polysaccharides (SSPS), and beta-glucan (BG). The texture and microstructure of different samples were analyzed. Antigenicity and IgE-binding capacity were determined using ELISA. Finally, peptide sequencing was used to identify the degradation degree and frequency of allergenic epitopes. Results: Samples with added PDX (F-PDX) and IN (F-IN) exhibited lower hardness; smaller, irregular pores; and a honeycomb structure, whereas samples with SSPS (F-SSPS) and BG (F-BG) had higher hardness; large, regular pores; and strong sheet structures. The antigenicity and IgE-binding capacity of F-PDX and F-IN were lower both before and after 120 min of in vitro dynamic gastrointestinal digestion. The peptidomics results indicated that F-PDX and F-IN primarily facilitated the degradation of the glycinin G1 and G2 subunits, beta-conglycinin alpha, and the beta subunit. Moreover, it increased the frequency of destruction of allergenic epitopes, and further promoted the degradation of epitopes in the external alpha-helix structures of glycinin and beta-conglycinin compared to FSPI. Conclusions: The addition of polysaccharides had a significant impact on the structure and allergenicity of the soy protein gel, especially PDX and IN.