Noncollinear antiferromagnetic order and effect of spin-orbit coupling in spin-1 honeycomb lattice

Motivated by the recently synthesized insulating nickelate Ni2Mo3O8, which has been reported to have an unusual noncollinear magnetic order of Ni2+ S = 1 moments with a nontrivial angle between adjacent spins, we construct an effective spin-1 model on the honeycomb lattice, with the exchange parameters determined with the help of first-principles electronic-structure calculations. The resulting bilinear-biquadratic model, supplemented with the realistic crystal-field induced anisotropy, favors the collinear Neel state. We find that the crucial key to explaining the observed noncollinear spin structure is the inclusion of the Dzyaloshinskii-Moriya (DM) interaction between the neighboring spins. By performing variational mean-field and linear spin-wave theory (LSWT) calculations, we determine that a realistic value of the DM interaction D approximate to 2.78 meV is sufficient to quantitatively explain the observed angle between the neighboring spins. We furthermore compute the spectrum of magnetic excitations within the LSWT and random-phase approximation, which should be compared to future inelastic neutron measurements.