On linear transform design with non-linear approximation

In this paper we share our recent observations on methods for sparsity enforced orthogonal transform design. In our previous work on this problem, our target was to design transforms (sparse orthonormal transforms - SOT) that minimize the overall sparsity-distortion cost of a collection of image patches mainly for improving the performance of compression methods. In this paper we go one step further to understand why these transforms achieve better approximation and how different they are from transforms like the DCT or the Karhunen-Loeve transform (KLT). Our study lead us to mathematically validate that for a Gaussian process the KLT is the optimal transform not only in a linear approximation sense but also in a nonlinear approximation sense, the latter forming the basis for sparsity-based regularization. This means that the search for SOTs yields the KLT in Gaussian processes, but results in transforms that are distinctly different from the KLT in non-Gaussian cases by capturing useful structures within the data. Both toy examples and real compression results in various representation domains are presented in this paper to support our observations.