Based on the granular micromechanical approach and energy conservation principle, an anisotropic micromorphic constitutive relationship has been obtained for granular materials. It is noted that the particle displacement induced by particle rotation has been ignored. Furthermore, the expressions of anisotropic constitutive tensors are derived through a recursive formula of the unit contact direction integral. On this basis, according to Hamilton's principle, the motion balance equation and boundary conditions of the anisotropic granular material are derived, and the dispersion relationship of the plane wave can be obtained for the anisotropic granular material. Finally, a detailed parameter analysis is carried out on the dispersion relationship and the frequency band gap. The research shows that: (1) The proposed model predicts that there are three types of waves in the granular materials, which consists of three kinds of longitudinal waves, six kinds of transverse waves and three kinds of in-plane transverse shear waves. For the transverse isotropic condition, the larger the value of the anisotropic parameter a(20) is, the higher the frequency of the longitudinal wave and the transverse wave are, and the lower the frequency of the in-plane transverse shear wave is. For the orthogonal anisotropy conditions, with the increase of the anisotropic parameter a(22), the frequency of transverse waves corresponding to the kinematics related to the 2-direction increases, whereas the frequency of transverse waves corresponding to the kinematics related to the 3-direction decreases. However, the coefficient a(22) has minor effects on the longitudinal wave. (2) The degree of fabric anisotropy has minor effects on the bandwidth of the transverse wave, but it has a greater effect on the bandwidth of the longitudinal wave: the increase of a(20) reduces the bandwidth between acousto-optical waves, whereas the bandwidth between optical waves increases. When a(20) is greater than 0.84, the band gap between acousto-optical waves disappears. In contrast, the increase of a(22) increases the bandwidth between acoustic and optical waves, whereas the bandwidth between optical waves decreases. When simplified to consider the isotropic condition, the dispersion curves of the three types of waves predicted by the proposed model show a good agreement with other benchmark theories.