Recent studies have shown that high-quality code semantics learning can effectively improve the performance of code clone detection. However, existing approaches suffer from two major drawbacks: (a) insufficient utilization of code representations, leading to inefficient semantics learning, and (b) low efficiency of clone detection, resulting in massive detection time. Therefore, we are motivated to propose an efficient semantics learning method while speeding up the detection process. Specifically, to address the first one, we adopt either CFG (Control Flow Graph) or PDG (Program Dependency Graph) as our initial code representation because of their rich semantic information. Further, we propose a novel graph-based code semantics learning method, which can capture critical information at token, statement, edge, and graph levels. To address the second one, we design a Siamese graph-matching network based on attention mechanisms. It can uniformly generate graph embeddings for code fragments and facilitate parallel detection of semantic clones, thus significantly boosting the speed of semantic clone detection.We evaluated our approach on two Java benchmark datasets, Google Code Jam and BigCloneBench. The experimental results show that our model outperforms the SOTA (State-Of-The-Art) lightweight models and is over 20x faster in detection. In addition, our model performs on par with the large Bert-based models and is over 110x faster in detection. Our code and dataset are available online at: https://github.com/HduDBSI/ CodeGraph4CCDetector.