Fermi surface instabilities of symmetry-breaking and topological types on the surface of a three-dimensional topological insulator

The emergence of the Pomeranchuk instability (PI) in a helical Fermi liquid (HFL) residing on the surface of a three-dimensional topological insulator (3D TI) is addressed at the mean-field level. An expression for the PI condition is derived in terms of a few microscopic parameters in each angular momentum channel corresponding to a central interaction between the helical electrons. It is found that because of the presence of strong spin-orbit coupling (SOC), the Landau parameter (F) over bar (l) corresponding to a particular angular momentum channel l depends not only on the electron-electron interaction in the same channel, but also interactions in (l + 1) and (l - 1) channels. The formalism automatically excludes the l = 1 PI in the HFL where the Galilean invariance is broken because of the presence of strong SOC. It is also found that the competing PIs can only be avoided until the appearance of l = 2 PI. In this case, the corresponding nematic instability can even be achieved in the l = 1 angular momentum channel of interaction between the electrons. The range of interaction between the electrons plays a pivotal role in bringing out the PIs. This is established by analyzing a few realistic profiles of the interaction. Another class of instability, involving a change in the topology of the Fermi surface without breaking the rotational symmetry, is found which competes with the PIs. Quantum phase transition originating from this instability is quite similar to the Lifshitz transition but is driven by electron-electron interaction. Possible connections of this instability with experiments are also described briefly.