PSK versus QAM for iterative decoding of bit-interleaved coded modulation

Bit-interleaved coded modulation with iterative decoding (BICM-ID) is a transmission scheme with excellent performance on fading channels. The mapping of the bit patterns to the elements of the signal constellation set (SCS) is the key design element of BICM-ID, especially regarding the asymptotic performance of the BER. In this paper we will demonstrate that SCSs of PSK (phase shift keying) with optimum bit pattern assignment are significantly superior in the asymptotic error-floor region to the respective square QAM (quadrature amplitude modulation) SCSs. When modifying the PSK SCSs to non-regular SCSs performance gains more than 1 dB are realizable in case of 4 bits per channel symbol. The achievable gains are theoretically determined and analyzed, and verified by simulations.