Linear transceive filters for relay stations with multiple antennas in the two-way relay channel

This paper considers the two-hop relaying case where two nodes S1 and S2 in a wireless network can communicate with each other via an intermediate relay station (RS) which is equipped with multiple antennas and cannot transmit and receive simultaneously on the same channel resources. In one-way relaying, four orthogonal channel resources are required for the transmissions from S1 to S2 and from S2 to S1. MIMO two-way relaying has been introduced as an approach which requires only half the channel resources compared to one-way relaying due to the simultaneous transmission from S1 to S2 and vice versa. For MIMO two-way relaying, a spatial filter matrix is required at the RS which applies both, transmit and receive processing. The design of linear spatial filter matrices, termed transceive filter matrices, is given in this paper. In particular, linear transceive filters are derived which fulfill the zero forcing (ZF) and minimum mean square error (MMSE) criterion, respectively. It is shown that the linear MMSE transceive filter outperforms the linear ZF transceive filter in terms of overall bit error rate (BER). However, for different channel qualities on the two channels to the RS, the choice of the transceive filter influences which direction of communication has a better BER performance.