Improving Physical Layer Security in Two-Way Relay Systems

This paper studies the physical layer security in a two-way relay system (TWRS) consisting of two transceivers, one eavesdropper, and multiple relays when the channel state information (CSI) of the eavesdropper is available. We consider the problem of maximizing the secrecy sum rate by optimizing the relay beamforming vector under relays' total power constraint. The optimization problem has been formulated in the literature as a product of three Rayleigh quotients, which is very difficult to solve, and the best proposed solution was a suboptimal solution to the null space beamforming. Here, we propose two approaches: 1) Optimal solution to the null space beamforming approach, and 2) Ignoring one Rayleigh quotient (IORQ) approach (which has not been proposed before). In the first approach, we convert the nonconvex product of two Rayleigh quotients into a convex problem with one dimensional search using semidefinite programming (SDP). Then we significantly reduce the complexity of solving the problem using the generalized eigenvalues. In the second approach, we look for the beamforming vector that does not eliminate the complete information signal at the eavesdropper aiming to increase the whole secrecy sum rate. Numerical results show that the proposed algorithms provide higher secrecy sum rates than the existing algorithms.