We consider small perturbations around the self-dual Chern-Simons as well as Maxwell Chern-Simons vortices. For the topological case, we show that there exist perturbations for which, at least to leading order, the vortices remain non-interacting. On the other hand, for the non-topological case, we show that for most of the perturbations the vortices are unstable against decay to the elementary excitations.