Design of fractional-order variants of complex LMS and NLMS algorithms for adaptive channel equalization

Equalization filtering is an effective technique applied to minimize the inter-symbol interference (ISI) in multipath fading channels; the problem gets worse for higher-order constellations which are required for high data rates in today’s communication systems. The least mean square (LMS) filter is a computationally efficient and easily implementable algorithm but suffers from slow convergence; highly complex filters are required to nullify the effects of ISI. In this paper, we develop complex modified fractional-order (FO) nonlinear variants of the LMS and the NLMS algorithms and apply in adaptive channel equalization, in both feed-forward and decision feedback configurations. In addition to the standard first-order derivative, the update in the modified LMS also depends on the FO derivative of the mean square error, the final update is formed using a combination of conventional update term and a nonlinear term obtained through Riemann–Liouville fractional derivative. The step size of the FNLMS scheme in fractional part is not only a function of the input energy but also the FO. The differintegral operator working as differentiator helps improve the convergence rate because the algorithm becomes nonlinear; the fractional algorithms provide more parameters to control the rate of convergence and have simple implementation with almost similar complexity. The performances of the schemes are validated through extensive simulation results for block fading channels (frequency flat and selective) to evaluate the symbol error rate for higher-order quadrature amplitude modulation schemes, mean square error and combined channel and equalizer responses to show the improved inverse modeling of the channel. Simulation experiments confirm the superiority of the proposed algorithm over the traditional counterparts.