Transceiver Design for Multi-user Multi-antenna Two-way Relay Channels

In this paper, we design transceivers in a multiuser multi-antenna two-way relay system, where a single multi-antenna base station exchanges information with multiple users via a single multi-antenna relay station. We consider the half-duplex amplify-and-forward relay protocol. We aim to maximize the bidirectional sum rate under the constraint of no interference among different users. Suboptimal solutions to the problem that respectively maximizing the uplink and downlink rates are derived. We then introduce a threshold to balance the uplink and downlink rates so as to maximize the bidirectional sum rate. Simulation results show that the proposed scheme achieves considerably higher bidirectional sum rate than existing schemes.