Joint precoding algorithm for multi-user two-way AF MIMO relay system

Precoding design for two-way amplify-and-forward (AF) multiple input multiple output (MIMO) relay systems has attracted more and more research interest recently. However, there is little research for multi-user two-way AF MIMO relay systems, which has important significance for the practical communication systems. A novel precoding design for multi-user two-way AF MIMO relay systems is investigated in which the minimum sum mean square error (MSMSE) is taken as the design criterion to solve the non-convex optimization problem, e. g. joint source, relay, and multi-user with power constraints. The initial non-convex optimization problem is transformed into multiple sub-optimization problems and the semi-definite programming (SDP) design, quadratically constrained quadratic programming (QCQP) design and second-order cone programming(SOCP) design are used to solve the sub-optimization problems. Afterwards, the local optimization value of each sub-convex optimization problem is further solved by alternate iterative methods to achieve the optimization of all node matrix variables. The proposed algorithm is more comprehensive and practical than the existing ones in terms of the optimization problems, and the simulation results show that the proposed iterative algorithm can significantly improve the sum mean square error (SMSE) performance, sum-rate and bit-error rate(BER) of the systems. © 2020, Editorial Office of Systems Engineering and Electronics. All right reserved.