[Significance] Two-phase flow instability is a classic problem in the field of steam generators and other two-phase flows. Therefore, it has been studied extensively. In nuclear reactor steam generators, two-phase flow instability may occur on the secondary side and interfere with the control system, causing fatigue-induced damage to the equipment. While two-phase flow instability can have different complex mechanisms and many influencing factors, there are various methods to research and analyze this phenomenon. [Progress] The phenomenon of two-phase flow instability can be classified into two types. The mechanisms of flow excursion (LE), density wave oscillation (DWO) and pressure drop oscillation (PDO) are introduced. LE and PDO can occur in conditions corresponding to the region of negative slope in the hydrodynamic characteristic curve (mass flow rate vs. pressure drop curve) in the heated tube and can be avoided by eliminating the negative slope region. However, DWO can also occur in the positive slope region due to the phase difference between two transient processes. One of these is the mass flow rate variation caused by variation in the driving pressure difference, which is controlled by the rate of momentum transfer. The other is the transient variation of the subcooled water region length and the density of saturated two-phase region fluid, which is caused by heat transfer. Changes caused by heat transfer are slower than changes in flow and pressure. Various research methods of two-phase flow instability are systematically summarized, including the theoretical time-domain method (nonlinear and linear methods), theoretical frequency-domain method, and discrete numerical method, starting from the conservation equations. The mathematical criterion obtained from the theoretical time-domain model can analyze the parameters' influence exactly over a wide range. The spatial distribution of density, enthalpy, and other physical parameters in the frequency domain can be obtained using the theoretical frequency-domain method, and the stability boundary it predicts is more accurate than that predicted by the theoretically simplified linear time-domain method. In addition, the research status of LE, DWO, and PDO is systematically summarized, with a particular focus on the work of our research group. New dimensionless numbers (two-phase number, superheated number, dimensionless pump number, and dimensionless bypass number) are proposed to describe the stability of the complex, superheated, two-phase flow boiling systems. A law unifying the influence of the Froude number, friction number, and geometric parameters (tube length, tube diameter, etc.) on DWO was developed. Previous contradictory conclusions are explained. A rigorous theoretical derivation and proof of the effects of model simplification and boundary conditions are presented. The requirements for conservatively modeling a real nuclear power plant steam generator and secondary loop system using a test section consisting of a single or multiple parallel small-scale heated tubes and a simplified engineering verification test loop in the laboratory are clarified. Finally, methods to avoid LE and DWO in the steam generator of the high-temperature gas-cooled reactor are introduced based on reactor design. To predict the stability of the high-temperature gas-cooled reactor-pebble bed module (HTR-PM) engineering test facility-steam generator (ETF-SG), theoretical time-domain method, theoretical frequency-domain method, RELAP5 model, and one-dimensional transient program are developed, which are in good agreement with the experiments. [Conclusion and Prospects] The results from the ETF-SG can conservatively predict the stability boundary of the steam generator and secondary loop of the HTR-PM nuclear power plant. The conditions for the occurrence of in-phase and out-of-phase DWO in ETF-SG are revealed, and methods for eliminating them are recommended. The above achievements are applied in the design, commissioning, and operation of the HTR-PM steam generator. © 2023 Press of Tsinghua University. All rights reserved.