Reducing carbon emissions from the transport sector is essential for realizing the carbon neutrality goal in China. Despite substantial studies on the influence of urban form on transport CO2 emissions, most of them have treated the effects as a linear process, and few have studied their nonlinear relationships. This research focused on 274 Chinese cities in 2019 and applied the gradient-boosting decision tree (GBDT) model to investigate the nonlinear effects of four aspects of urban form, including compactness, complexity, scale, and fragmentation, on urban transport CO2 emissions. It was found that urban form contributed 20.48% to per capita transport CO2 emissions (PTCEs), which is less than the contribution of socioeconomic development but more than that of transport infrastructure. The contribution of urban form to total transport CO2 emissions (TCEs) was the lowest, at 14.3%. In particular, the effect of compactness on TCEs was negative within a threshold, while its effect on PTCEs showed an inverted U-shaped relationship. The effect of complexity on PTCEs was positive, and its effect on TCEs was nonlinear. The effect of scale on TCEs and PTCEs was positive within a threshold and negative beyond that threshold. The effect of fragmentation on TCEs was also nonlinear, while its effect on PTCEs was positively linear. These results show the complex effects of the urban form on transport CO2 emissions. Thus, strategies for optimizing urban form and reducing urban transport carbon emissions are recommended for the future.