Optimized prototype filter based on the FRM approach for cosine-modulated filter banks

A design procedure for frequency-response masking (FRM) prototype filters of cosine-modulated filter banks (CMFBs) is proposed. In the given method, we perform minimization of the maximum attenuation level in the filter's stopband, subject to inter-symbol interference (ISI) and intercarrier interference (ICI) constraints. For optimization, I a quasi-Newton algorithm with line search is used, and we provide simplified analytical expressions to impose the interference constraints, which greatly reduce the computational complexity of the optimization procedure. The result is lower levels of ISI and ICI for a predetermined filter order, or a reduced-filter complexity for given levels of interferences. It is then illustrated how the FRM-CMFB structure is suitable for implementing filter banks with a large number of bands, yielding sharp transition bands and small roll-off factors, which is an attractive feature for a wide range of practical applications.