Objective As a lightweight fiber-reinforced composite material with high strength and modulus, ultra-high molecular weight polyethylene (UHMWPE) plays an important role in modern aviation, aerospace, marine defense equipment, and other fields. Because its smooth surface is chemically inert and does not contain polar groups, the poor binding with the resin matrix would affect its interface performance. The chemical composition reconstruction on the surface of UHMWPE fiber was carried out by oxygen plasma modification, which effectively provided more polar sites for the interface layer to improve the interface binding performance between UHMWPE fiber and resin matrix. Method In order to investigate the influence of damage mode on interlayer fracture toughness of oxygen plasma modified UHMWPE/vinyl ester composites. UHMWPE plain fabrics with a weft density of 200 picks/(10 cm) and warp density of 160, 200 and 240 picks/(10 cm) were woven and modified by oxygen plasma. UHMWPE/vinyl ester composites were prepared by vacuum-assisted resin infusion molding (VARI). Mode I (GⅠC) and mode Ⅱ (GⅡC) interlayer fracture toughness tests were carried out. The damage patterns were analyzed by the acoustic emission (AE) technique. Results After oxygen plasma modification of UHMWPE fabrics, the water contact angle of the fiber surface was found to be reduced (Fig. 2), improving surface energy. For GⅠC, the maximum loads of GⅠC of samples U03 and P03 were 19.49 and 27.05 N, respectively. It was found that the plasma modified GⅠC of sample P03 by 36.8%, and the mode Ⅰ interlayer fracture angles of samples P01 and P03 were 26° and 10°, respectively (Fig. 3). P01 displayed the greatest change in interlayer fracture angle. The GⅠC of UHMWPE/vinyl ester composites was mainly related to the interfacial bonding properties, the woven fabrics structure, the state of the pre-crack tip, and the brittleness of the matrix. During the GⅠC test, AE cumulative energy of UHMWPE/vinyl ester composites increased as the warp density of UHMWPE fabric increased (Fig. 4). Because of the influence of the compact structure of UHMWPE fabric on the uniformity of modification, the interface bonding appeared to be poor. The three damage modes in the process of the GⅠC test responded to acoustic emission of three types of signals, respectively, i.e. Class-1 signal indicating matrix cracking, Class-2 signal fiber/matrix debonding, and Class-3 signal fiber fracture (Fig. 5). After oxygen plasma modification, the intensity of the Class-2 signal was reduced, suggesting improvement in the fiber/matrix bonding. For GⅡC, the maximum loads of GⅡC of samples U03 and P01 were 106.99 and 244.58 N. The plasma modification improved GⅡC of sample U01 by 1 120%, and the minimum crack propagation length of sample P03 became 0.8 mm. The maximum bending fracture angle was 145° (Fig. 7). Sample P01 showed good GⅡC after oxygen plasma modification, and sample P03 demonstrated the smallest crack propagation length and flexural fracture angle. The GⅡC of UHMWPE/vinyl ester composites was mainly related to the interfacial bonding properties, stiffness, the state of the initial crack tip, and the brittleness of the matrix. During the GⅡC test, UHMWPE/vinyl ester composites modified by oxygen plasma delayed the start time of AE as cumulative energy, and different damage modes were reduced or eliminated (Fig. 8). The three AE signals generated by the damage modes during the GⅡC test were the same as those in the GⅠC test (Fig. 9). After oxygen plasma modification, the signal intensity of Class-3 in the sample was reduced, and the fiber fracture was reduced. Conclusion UHMWPE/vinyl ester composites are prepared by oxygen plasma modification of UHMWPE fabrics with different warp densities, and GⅠC and GⅡC were evaluated. The oxygen plasma modified UHMWPE/vinyl ester composites can effectively strengthen the GⅠC and GⅡC, improve the interface bonding strength and prevent crack propagation. For the loose structure of UHMWPE fabric with low warp density, the uniformity of oxygen plasma modification is the best, which has the effect of interlayer toughening. Three damage modes of UHMWPE/vinyl ester composites in GⅠC and GⅡC tests are detected by acoustic emission testing, which are matrix cracking, fiber/matrix debonding, and fiber fracture. The overlapped signals of the three damage modes in the amplitude distribution interval indicate that the damage does not appear alone, and the appearance of new damage is usually accompanied by the expansion of the previous damage, leading to the simultaneous appearance of different damage modes. Oxygen plasma modification can effectively weaken or eliminate interlayer damage and inhibit interlayer crack propagation. © 2023 China Textile Engineering Society. All rights reserved.