Defects widely exist in rock masses, which have great impacts on the rock mechanical properties. The pre-existing fissures and holes are two common rock defects. In order to study the influence of hole shapes and fissure geometries on the failure modes and mechanical properties of rock masses, the derivatives of kernel functions in the traditional SPH method have been improved by introducing a fracture mark ξ, and a novel numerical method named improved kernel of smoothed particle hydrodynamics (IKSPH) has been developed to realize the simulations of rock fracture processes. The Monte Carlo method has also been introduced into the proposed method to realize the formations of random fissures. The small-scale numerical rock specimens with different hole shapes as well as random fissures are established. The crack propagation processes, the failure modes and the mechanical properties under uniaxial compression are simulated, results show that: ‘Wing cracks’ initiate from the random fissures, whose locations and interaction modes differ according to the geometries of random fissures. For rectangle hole, the crack initiations mainly occur at the upper and lower edges as well as the corners; for circular hole, tensile cracks appear at the 6 and 12 o’clock directions, meanwhile, shear cracks initiate at the 5 and 7 o’clock direction after the tensile crack propagates to a certain extent; for trapezoid and triangle hole, crack initiations mostly appear at the corners of the hole; the length and quantity of pre-existing fissures have greater influences on the mechanical properties of rock masses than the hole shapes and fissure dip angles. The research results can provide guidance for the correct understandings of fractured rock mechanisms. Meanwhile, the proposed method can also provide references for the applications of SPH method into simulations of rock fracture mechanics. At the same time, future research focus should be put on the applications of IKSPH method into engineering practices and developing high-performance IKSPH programming.