Fast guided filter for power-efficient real-time 1080p streaming video processing

With the advent of embedded vision systems, smart sensors with integrated image signal processing (ISP) become a hot topic. This poses a need for efficient hardware implementation, regarding resource utilization and power consumption, of core image processing algorithms. Power consumption is especially important, since many of the target devices are usually battery operated. Edge-aware filtering, although it is used in many core image processing algorithms, is still challenging operation, especially in cases where large kernels are needed. In this paper, efficient hardware realization of fast guided filter (FGF) is proposed. It is based on idea that large filter of size R=K·S\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R=K \cdot S$$\end{document} can be calculated by downsampling input image by factor S and using filter of size K. Besides reduced memory and logic requirements, this optimization enables that, for the scaling factor S, core processing is done at 1/S2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1/S^{2}$$\end{document} pixel clock, providing significantly lower power consumption. Experimental results on Cyclone V FPGA chip demonstrate that, for FGF of size 35×35\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$35 \times 35$$\end{document} with downsampling factor S=7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S=7$$\end{document}, the proposed design achieves 60 fps for 1080p video. Memory utilization is 147.3 kB without need for any off-chip memory. Core dynamic power consumption is 79.89 mW. Proposed design consumes less total power than state-of-the-art guided filter realizations including ASIC-based solutions. This module can be seamlessly integrated into smart sensors ISP units, because it is designed for power-efficient streaming processing.