The current classification of turbulent boundary layers over different wall surfaces is based on the effect the roughness has on the mean velocity, via the roughness function ΔU+. Previous hot-wire measurements have shown that turbulent boundary layers over different rough wall surfaces, but with identical ΔU+, contain significant differences in the Reynolds stresses throughout the layer. This suggests that a detailed documentation of the effect of the roughness on the anisotropic Reynolds stress tensor may pave the way for a more general classification. This paper examines experimental (boundary layer) and direct numerical simulation (channel flow) data for the invariants of the Reynolds stress tensor. Although only a limited number of rough surfaces have been examined, the results indicate that, relative to a smooth wall, the roughness reduces the level of anisotropy. This is more prominent for k-type roughnesses, the anisotropic invariant map (AIM) signature of the d-type roughness being closer to that for a smooth surface. In the vicinity of the roughness, the AIM signature varies dramatically, within one roughness wavelength, reflecting the significant changes in the characteristics of the turbulence field.