We have investigated the initial oxynitridation of an atomically flat Si(001)-2x1 surface by NO. We found that appropriate oxynitridation conditions, in which oxide decomposition does not occur, are required to suppress the roughness of the NO-reacted Si surface. Under these conditions, we investigated the growth behavior and chemical structure. The first oxynitridation, in which NO reacts with the first Si layer on the Si(001)-2x1 surface, takes place in a layer-by-layer manner caused by two-dimensional nucleation. However, further oxynitridation for the second Si layer proceeds in a three-dimensional manner in which the atomic-scale roughness at the oxynitride/Si interface increases. In addition, under our oxynitridation conditions, N is incorporated as Ni=Si-3, even though the oxynitride is ultrathin.