PERFORMANCE ANALYSIS OF A MODIFIED RAO TEST FOR ADAPTIVE SUBSPACE DETECTION

The problem of detecting a subspace signal is studied in colored Gaussian noise with an unknown covariance matrix. In the subspace model, the target signal belongs to a known subspace, but with unknown coordinates. We propose a modified Rao test (MRT) by introducing a tunable parameter. The MRT is more general, which includes the Rao test and the generalized likelihood ratio test as special cases. Moreover, closed-form expressions for the probabilities of false alarm and detection of the MRT are derived. Numerical results demonstrate that the MRT can offer the flexibility of being adjustable in the mismatched case where the target signal deviates from the presumed signal subspace. In particular, the MRT provides better mismatch rejection capacities as the tunable parameter increases.