Shear strength parameters play a crucial role in the design and construction of rock slopes, underground openings, tunnels, excavations, and foundations. Determination of cohesion (c) and angle of internal friction (phi) through laboratory tests (e.g., triaxial tests) and in situ tests is time-consuming and expensive; therefore, it is valuable to accurately predict the shear strength parameters of rock materials using less expensive and more reliable methods. In this study, 4 artificial intelligence models, namely group method of data handling (GMDH), gene expression programming (GEP), bidirectional long short-term memory network (BILSTM), and random forest (RF), are proposed to predict the rock shear strength parameters c and phi. A database of 199 sets of experimental data from 4 rock types was used to construct the proposed models. The results show that the accuracy of the RF model is superior to the other three models, with coefficients of determination of 0.9933 (c) and 0.9727 (phi) on all datasets, respectively. To further improve the prediction accuracy of the RF model, six metaheuristic algorithms, namely particle swarm optimization (PSO), bald eagle search (BES), marine predators algorithm (MPA), northern goshawk optimization (NGO), golden jackal optimization (GJO), and dung beetle optimizer (DBO), were used to optimize the hyperparameters of the model. The results show that the accuracy of the six hybrid RF models is higher than that of the single RF model. Among the 6 hybrid RF models, the hybrid DBO-RF model is superior to the other hybrid RF models. In addition, the factors affecting the shear strength of rock materials are analyzed through the parameter sensitivity analysis. Four artificial intelligence models are used to predict the shear strength parameters of rock materials.Six metaheuristic algorithms are used to optimize the hyperparameters of the random forest model.The prediction performance of the six hybrid random forest models is compared.The factors influencing the shear strength of rock materials are analyzed by parameter sensitivity analysis.