Context. Turbulent small-scale structures in the envelopes and winds of massive stars have long been suggested as the cause for excessive line broadening in the spectra of these stars that could not be explained by other mechanisms such as thermal broadening. However, while these structures are also seen in recent radiation-hydrodynamical simulations, their origin, particularly in the envelope, has not been extensively studied. Aims. We study the origin of structures seen in 2D radiation-hydrodynamical unified stellar atmosphere and wind simulations of O stars and Wolf-Rayet stars. Particularly, we study whether the structure growth in the simulations is consistent with sub-surface convection, as is commonly assumed to be the origin of this turbulence. Methods. Using a linear stability analysis of the optically thick, radiation-pressure dominated envelopes of massive stars, we identified multiple instabilities that could be driving structure growth. We quantified the structure growth in the non-linear simulations of O stars and Wolf-Rayet stars by computing density power spectra and tracking their temporal evolution. Then, we compared these results to the analytical results from the stability analysis to distinguish between the different instabilities. Results. The stability analysis leads to two possible instabilities: the convective instability and an acoustic instability that is a local variant of so-called strange modes. Analytic expressions for the growth rates of these different instabilities are found. In particular, strong radiative diffusion damps the growth rate omega of the convective instability in this regime leading to a distinct omega similar to 1/k(2) dependence on wavenumber k. From our power spectra analysis of the simulations, however, we find that structure growth rather increases with k - tentatively as omega similar to root k. Conclusions. Our results suggest that, contrary to what is commonly assumed, structures in luminous O and Wolf-Rayet star envelopes do not primarily develop from the sub-surface convective instability. Rather the growth seems compatible with either the acoustic instability in the radiation-dominated regime or with Rayleigh-Taylor type instabilities, although the exact origin remains inconclusive for now.
