MMSE-based joint source and relay optimization for interference MIMO relay systems

In this paper, we investigate the transceiver design for amplify-and-forward (AF) interference multiple-input multiple-output (MIMO) relay communication systems when the direct links between the source and destination nodes are taken into consideration. The minimum mean-squared error (MMSE) of the signal waveform estimation at the destination nodes is chosen as the design criterion to optimize the source, relay, and receiver matrices for interference suppression. As the joint source, relay, and receiver optimization problem is nonconvex with matrix variables, a globally optimal solution is computationally intractable to obtain. We propose two iterative algorithms to provide computationally efficient solutions to the original problem through solving convex subproblems. These two algorithms provide efficient performance-complexity trade-off. Simulation results demonstrate that the proposed algorithms converge quickly after a few iterations and significantly outperform existing scheme in terms of the system bit error rate.