Optomechanical backaction in the bistable regime

With a variety of realizations, optomechanics utilizes its light-matter interaction to test fundamental physics. By coupling the phonons of a mechanical resonator to the photons in a high-quality cavity, control of increasingly macroscopic objects has become feasible. In such systems, state manipulation of the mechanical mode is achieved by driving the cavity. To be able to achieve high drive powers the system is typically designed such that it remains in a linear response regime when driven. A nonlinear response, and especially bistability, in a driven cavity is often considered detrimental to cooling and state preparation in optomechanical systems and is avoided in experiments. Here we show that with an intrinsic nonlinear cavity backaction cooling of a mechanical resonator is feasible operating deep within the nonlinear regime of the cavity. With our theory taking the nonlinearity into account, precise predictions on backaction cooling can be achieved even with a cavity beyond the bifurcation point, where the cavity photon number spectrum starts to deviate from a typical Lorentzian shape.