Quantum phase transition in the three-dimensional anisotropic frustrated Heisenberg antiferromagnetic model

The phase diagram of the anisotropic quantum spin-1/2 Heisenberg antiferromagnet in the presence of competitive nearest (J(1))- and next-nearest (J(2) = alpha J(1)) -neighbor interactions on a simple cubic lattice is studied within the framework of the differential operator technique. The Hamiltonian is solved by using an effective-field theory in a two-spin cluster. We propose a function for the free energy, which is similar to the Landau expansion, in order to find the ground state (T=0) and the finite-T phase diagram. At zero temperature, we find a first-order quantum phase transition in the point alpha(c)(Delta) between the antiferromagnetic (AF) and superanti-ferromagnetic (SAF) phases. This is dependent on the anisotropy parameter Delta (Delta=0 and 1 correspond to the isotropic Heisenberg and Ising models, respectively). The quantum critical values of alpha c(Delta not equal 1)=alpha(q)(c)(Delta) are smaller than those of the classical model alpha(c)(Delta=1)=alpha(c)(c)=1/4 (for example, for Delta=0, we find alpha(q)(c)(0) =0.21). The former monotonically increases with the anisotropy parameter Delta. The phase diagram in T-alpha plane presents a second-order transition at T-2(alpha) between the AF and paramagnetic (P) phases. The phase transition between the SAF and P phases is of first-order T-1(alpha) for all values of Delta is an element of [0, 1]. The phase transition lines T-1(alpha) and T-2(alpha) merge in a critical end point (CEP) that is dependent on Delta [i.e., T-CEP(Delta) and alpha(CEP)(Delta) are the temperature and frustration parameters at the CEP]. The ordered phases (AF and SAF) are separated by first-order transitions and exhibit a nontrivial re-entrant behavior at low temperatures.