The gyrotron traveling wave tube (gyro-TWT) is a high-power, broadband, millimeter-wave amplifier. This study employed a nonlinear self-consistent code to analyze nonlinear oscillation behavior of a gyro-TWT. There are three operating regimes in the gyro-TWT, including amplification, nonlinear oscillation and mode competition regimes. Based on theory of nonlinear oscillation, the amplification and nonlinear oscillation modes are the stable modes and compete with each other in the mode competition regime. An oscillator plane of the gyro-TWT is elucidated in the paper. This work shows that the amplification mode transits to the nonlinear oscillation mode where the input power exceeds the critical power and the beam current is above the start-oscillation current on the free-running operation. Moreover, the peak gain for the nonlinear oscillation mode is enhanced as the frequency of the input sign approaches the oscillation frequency on free-running operation.