In order to solve the problems of inefficient allocation of teaching resources and inaccurate recommendation of learning paths in higher education, this paper proposes a smart education optimization model (SEOM) by combining the improved random forest algorithm (RFA) based on adaptive enhancement mechanism and the Graph Neural Network (GNN) algorithm. The public data and information such as the national higher education intelligent education platform are collected, and SEOM is trained and verified. The results show that SEOM has high accuracy and generalization ability in three different teaching scenes: online mixed teaching, personalized teaching and project-based teaching. The Root Mean Square Error (RMSE) value in cross-validation is between 0.2 and 0.5, and the Mean Absolute Error (MAE) value is between 0.1 and 0.5. SEOM shows strong stability when dealing with multidimensional educational resources and complex teaching modes. The accuracy rate remains at 85-97%, indicating its reliability in personalized learning path recommendation. Further analysis shows that the chi-square freedom ratio is between 1.0 and 2.5, the fitting index and the adjusted fitting index are both above 0.85, and the comparative fitting index is close to 0.95, which shows that SEOM has high accuracy and rationality in capturing the dependence of knowledge points in different teaching modes. The Root Mean Square Residual (RMR) and Root Mean Square Error of Approximation (RMSEA) are both below 0.05, which indicates that SEOM has small residual and strong scene adaptability. In addition, in the abnormal network environment, the resource allocation efficiency of SEOM is above 60%, and the Shapley value is between 0.1 and 0.4, which shows that SEOM can adapt to the change of network environment and the resource allocation effect is still obvious. Generally speaking, SEOM can optimize the allocation of educational resources and recommend learning paths in a complex environment, and effectively improve the intelligence and efficiency of teaching decision-making, especially for university administrators and educational technology developers.