Objective When a laser is transmitted in the atmosphere, atmospheric turbulence causes random fluctuations in the received light intensity, which seriously affects the performance of the free space optical (FSO) system. At the same time, the phase noise introduced by the electrolytic modulator module in the M-th quadrature amplitude modulation (MQAM) FSO system can cause the signal phase to shift, leading to degradation of demodulation performance and an increase in system error rate. Therefore, it is crucial to design a high-performance adaptive transmission scheme that can alleviate the severe degradation of optical signal transmission caused by atmospheric channels. The atmospheric turbulence effect in atmospheric channels is a slow fading process compared with high-speed coded modulation systems. By mapping the constellation points in the outer circle of the constellation map to a position close to the constellation origin with a certain probability and rules and sending and transmitting the mapped constellation points, probability shaping (PS) is achieved. Therefore, it is feasible to optimize the source distribution of the MQAM encoding FSO system and achieve adaptive adjustment of the transmitter signal constellation distribution based on channel conditions. In summary, we propose an adaptive PS system model in atmospheric channels that considers the combined effects of Gamma-Gamma turbulence and phase noise. This scheme can effectively improve the capacity of FSO communication links with QAM and suppress the impact of atmospheric turbulence on FSO systems. Methods We consider the comprehensive impact of atmospheric turbulence and phase noise of the electrolytic modulation module on the performance of optical communication systems. A model describing the combined effects of phase noise and atmospheric turbulence is constructed using Tikhonov distribution and Gamma-Gamma turbulence model. By iteratively searching through the particle swarm optimization algorithm in the heuristic algorithm, we obtain the optimal probability distribution under the condition of maximizing the decoding rate of the bit metric and realize rate adaptation with the bit metric decoding rate as the objective function through adaptive 16QAM PS technology. We also evaluate the error transmission performance of the system through pre-FEC BER and NGMI threshold before FEC decoding and investigate the effects of different turbulence intensities, phase noise, and fixed turbulence intensities with varying phase noise levels on the decoding rate, pre-FEC BER, and NGMI performance of contrast metrics. Results and Discussions A practical 16QAM encoding modulation FSO scheme with adaptive PS is proposed to address the issue of traditional channel coding techniques in FSO being susceptible to turbulence fading effects and the low achievable information rate of the system in conventional encoding uniform distribution mode. Compared with uniform distribution: the PS scheme can effectively improve the BMD rate under the combined influence of turbulence and phase noise; PS technology can improve bit error rate performance, and this improvement can be more pronounced in modulation schemes with lower entropy. The shaping gain between the PS scheme corresponding to the three information entropies and the uniform distribution under weak turbulence increases with the increase of BER. Conclusions We design a practical adaptive FSO coding modulation scheme based on PS. Due to the constraint of transmission rate, the optimization problem becomes non convex, increasing the complexity of the optimization process. Therefore, a heuristic algorithm is used to solve the optimization problem. The optimal probability distribution of the source is obtained by maximizing the bit metric decoding rate, and an adaptive PS signal-to- noise ratio threshold lookup table is established based on the optimal bit metric decoding rate switching point, greatly improving the capacity of the QAM FSO system. In addition, we also evaluate the NGMI and pre-FEC BER performance of the proposed scheme. The simulation results show that by considering the combined effects of phase noise and atmospheric turbulence, the adaptive PS scheme with non-uniform distribution can achieve a maximum shaping gain of 4. 8 dB compared with uniform distribution. Moreover, when the turbulence intensity is constant, the increase in phase noise has a greater impact on the selected PS distribution under low signal-to- noise ratio. Therefore, PS technology brings a better compromise between effectiveness and reliability for the performance of the 16QAM modulation format system in FSO communication.
