As the main component of recycled concrete, aggregate plays the role of skeleton and support, which has a great influence on the performance of recycled aggregate concrete, and the particle size of aggregate is one of the main factors affecting the mechanical properties of recycled aggregate concrete. In order to investigate the influence of aggregate particle size on the uniaxial compressive strength of recycled aggregate concrete, the uniaxial compression model of recycled aggregate concrete under different aggregate particle sizes and aggregate volume proportions was established by using discrete element numerical simulation method, and the mechanical characteristics and crack propagation mechanism of recycled aggregate concrete under uniaxial compression were simulated. The results show that the peak stress and elastic modulus of recycled aggregate concrete increase with the aggregate particle size, and the crack distribution is more uniform when aggregate particle size is 1.5 cm. With the increase of the proportion of large-particle size aggregate, the elastic modulus of recycled aggregate concrete gradually increases, and the peak stress first increases and then decreases. Obviously, the incorporation of moderate particle size coarse aggregate can improve the mechanical properties of the specimen. In the mixed particle size aggregate model, large, medium, and small particle sizes affect the crack generation and propagation of the contact surface, aggregate, and mortar, respectively. In the model with a volume ratio of 1:1:1, the peak strength is the highest and the crack distribution is the most uniform in the three phases.