The hydrodenitrogenation behavior differences between indole and quinoline were investigated using density functional theory calculation to explain the phenomena of previous experimental results. Theoretical calculation was carried out to explain the results using GGA-RPBE function with dispersion force correlation. The process of hydrogenation of indole, quinoline and 1,2,3,4-tetrahydroquinoline as well as denitrogenation of o-ethylaniline and o-propylaniline were conducted on a NiMoS nanocluster. The results suggest Ni-S-Edge is suitable for hydrogenation saturation of aromatic rings of nitrogen compounds and Ni-Mo-Edge is responsible for hydrogenolysis of C-N bond via E2 path. Besides, indoline could be directly converted to o-ethylaniline, whereas the C-N bond of 1,2,3,4-tetrahydroquinoline could only be broken after complete saturation of the aromatic ring. The reason is that the N atom and C=C bond of indoline could be coplanar and well adsorbed on Ni-Mo-Edge simultaneously, which is beneficial to reduce the activation energy of C-N bond cleavage.