This study aims to investigate the effect of fine aggregate lithology on the performance of rubberized asphalt mixtures. SEM, XRD, and contact angle tests were used to analyze the microscopic physical and chemical properties of 0-3-mm grain-size diabase and limestone. The performance characteristics of two types of ARAC-13 rubberized asphalt mixtures (with the diabase coarse aggregates and diabase fine aggregates, and diabase coarse aggregates and limestone fine aggregates, respectively) were then compared through conventional high-temperature water stability performance tests, uniaxial penetration tests, and accelerated loading abrasion tests. The results show that the surface roughness, chemical composition, and surface adhesion properties of limestone are better than those of diabase, and the surface roughness of limestone is 13.9% higher than that of diabase, and the surface adhesion energy is 6.4% higher. However, the results show relatively small differences between the asphalt mixtures with the diabase coarse aggregates and diabase fine aggregates and those with diabase coarse aggregates and limestone fine aggregates in the conventional water stability performance test and the high-temperature performance test. Furthermore, the strength reaches 90% when cured for 8 h for limestone and 12 h for diabase, with the curing rate being faster for limestone. The anti-abrasion performance of diabase specimens is superior to that of limestone specimens, but the anti-abrasion decay performance is the opposite. In summary, it can be observed that limestone fine aggregate shows better adhesion performance with asphalt than diabase fine aggregate, and it was recommended that diabase coarse aggregate and limestone fine aggregate be used to improve the asphalt-aggregate interface and enhance the road performance of the asphalt pavement.