On the performance and fiber nonlinearity modeling of learned digital back-propagation in the presence of DAC and ADC quantization noise

The noise and nonlinear signal distortion introduced by transceiver and signal propagation limit the achievable signal-to-noise ratio in coherent optical communication systems. As an efficient nonlinear compensation (NLC) scheme, digital back-propagation (DBP) algorithm is known to compensate for deterministic nonlinear interactions, but not for signal-noise interactions. In contrast, learned digital back-propagation (LDBP), which utilizing data-driven optimization based on physical models, offers what we believe to be a new approach to end-to-end compensation. In this paper, we investigate the impact of quantization noise due to the finite resolution of digital-to-analog converter and analog-to-digital converter on the performance of LDBP, and propose a modified analytical nonlinear interference (NLI) model by accounting the nonlinear beating between transceiver quantization noise and signal for LDBP. The split-step simulations show that the parameters of LDBP, which trained at an ideal link without quantization noise, demonstrate significant adaptability to the link with quantization noise. The results also indicate that the impact of quantization noise on LDBP can be less than that on DBP. Meanwhile, based on the simulations, the effectiveness of the proposed modified analytical NLI model is validated, and the results show excellent agreement between the proposed modified analytical model and the simulations, with discrepancies within 0.2 dB. (c) 2025 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement