In mining, dam foundations, underground caverns and petroleum engineering, engineers often face problems associated with jointed rock mass. The prime objective of this study is to improve our understanding of the shear behaviour of rock joints. Shear tests of several tensile rock joint samples under different normal loads have been conducted in order to relate the peak shear strength of a rock joint with the three-dimensional (3D) surface morphology. In the study, two parameters, which are termed as the average inclination angle (theta) over bar* and the maximum contact area ratio A(0), are used to characterise the surface of a rock joint, and the parameters can be easily measured using a morphology scanner system. A rational function of dilatancy angle is proposed based on the variation of the parameters which can be easily measured. Then, a new empirical peak shear strength criterion of Mohr-Coulomb type which has the capability of estimating the peak shear strength at the laboratory scale is proposed. Finally, a comparison among the proposed criterion, Barton's criterion, Grasselli's criterion and Xia's criterion are made from the perspective of both the reasonableness of the formula and the prediction accuracy. The advantages of the proposed criterion were analysed in detail. The proposed criterion was suitable for smooth horizontal rock joints. In addition, the dilatancy angle under zero and infinite normal stress was taken into consideration. Through the comparison, we could say the proposed criterion, which was easier and more intuitive from an engineering point of view, predicted the peak shear strength of rock joints with accurate precision.