Performance analysis of cooperative multiple-input multiple-output relaying with orthogonal space-time block codes (STBCs) in the presence of keyhole effects

Keyhole effect, under which a multiple-input multiple-output (MIMO) channel has uncorrelated spatial fading between antenna arrays but a rank-deficient transfer matrix, may exist in MIMO fading environments in realistic propagation environments. Keyhole effect will lead to significant performance degradations. Fortunately, recent researches demonstrate that cooperative diversity can mitigate keyhole effects. This study presents performance analysis of orthogonal space-time block coded transmission for a cooperative MIMO relaying system with Nakagami-m fading on both sides of the keyholes. In particular, decode-and-forward (DF) relaying, amplify-and-forward (AF) relaying, and DF-AF selection relaying over keyhole MIMO relay channels are, respectively, investigated. For DF relaying, exact outage probability and symbol error probability (SEP) are obtained. With respect to AF relaying and DF-AF selection relaying, the lower and upper bounds are derived. Theoretical analysis which has been further verified through Monte Carlo simulations demonstrate that in terms of outage probability and SEP over keyhole MIMO relay channels, DF-AF selection relaying is better than both AF relaying and DF relaying. Performance analysis