Trimer Formation and Metal-Insulator Transition in Orbital Degenerate Systems on a Triangular Lattice

As a prototypical self-organization in the system with orbital degeneracy, we theoretically investigate trimer formation on a triangular lattice, as observed in LiVO2. From the analysis of an effective spin-orbital coupled model in the strong correlation limit, we show that the previously proposed orbital-ordered trimer state is not the lowest-energy state for a finite Hund's-rule coupling. Instead, exploring the ground state in a wide range of parameters for a multiorbital Hubbard model, we find an instability toward a different orbital-ordered trimer state in the intermediately correlated regime in the presence of a trigonal crystal field. The trimer phase appears in the competing region among a paramagnetic metal, band insulator, and Mott insulator. The underlying mechanism is the nesting instability of the Fermi surface owing to a synergetic effect of Coulomb interactions and trigonal-field splitting. The results are compared with experiments in triangular-lattice compounds, LiVX2 (X = O, S, Se) and NaVO2.