In this letter, we present the generation, the peak-to average power ratio (PAPR) reduction, the heterodyne detection, the self-mixing reception, and the transmission performance evaluation of 16QAM-OFDM signals in 60 GHz radio over fiber (RoF) system using Discrete multitone (DMT) modulation and Better Than Nyquist pulse shaping (BTN-PS) technique. DMT modulation is introduced in the RoF system, in-phase and quadrature (IQ) will not be required using BTN-PS method, and the computation complexity is much lower than other published techniques for reduced PAPR in the RoF system. In the experiment, 5 Gb/s 16QAM-OFDM downlink signals are transmitted over 42 km SMF-28 and a 0.4 m wireless channel. The experimental results show that the receiver sensitivity is effectively enhanced using this method. Therefore, the introduced BTN-PS technique and its application is a competitive scheme for reducing PAPR, and enhancing the receiver sensitivity in future RoF system. (C) 2015 Elsevier Ltd. All rights reserved.