Absence of quantum anomalous Hall state in 4d transition-metal-doped Bi2Se3: An ab initio study

The realization of insulating ferromagnetic states in topological insulator (TI) systems, with sufficiently high Curie temperatures (T-C) and large magnetically induced gaps, has been the key bottleneck towards the realization of the quantum anomalous Hall effect (QAHE). Despite the limited reports on 3d or 4f transition-metal (TM)-doped Bi2Se3, there remains a lack of systematic studies on 4d TMs, which may be potential candidates since the atomic sizes of 4d TMs and that of Bi are similar. Here, we report a theoretical work that probes the magnetic behaviors of the 4d TM-doped Bi2Se3 system. We discovered that among the 4d TMs, Nb andMo can createmagnetic moments of 1.76 and 2.96 mu B in Bi2Se3, respectively. While Mo yields a stable gapless antiferromagnetic ground state, Nb favors a strong ferromagnetic order, with the magnetic coupling strength (T-C)similar to 6 times of that induced by the traditional Cr impurity. Yet, we found that Nb is still unfavorable to support the QAH state in Bi2Se3 because of the reduced correlation in the t(2g) band that gives a gapless character. This rationale is not only successful in interpreting why Nb, the strongest candidate among 4d TMs for achieving ferromagnetism in Bi2Se3, actually cannot lead to QAHE in the Bi2Se3 system even with the assistance of codoping but also is particularly important to fully understand the mechanism of acquisition of insulating ferromagnetic states inside TI. On the other hand, we discovered that Mo-doped Bi2Se3 favors strong antiferromagnetic states and may lead to superconducting states.