CNN Inference Accelerators with Adjustable Feature Map Compression Ratios

Recently, an increasing interest has been in developing a convolution neural network (CNN) with adjustable configurations, enabling instant adaption to different resource constraints during inference. The trained CNN in run-time can switch to different modes to achieve a certain accuracy-energy trade-off point, similar to DVFS (dynamic voltage and frequency scaling) and turbo boost, which are widely adopted in CPUs. In this paper, we propose strategies to enable CNN inference accelerators to have an adjustable feature map compression ratio, making them tunable regarding their external memory access amount. We resort to the mature JPEG technique to compress those intermediate feature maps. The critical challenge is to support such adjustable compression ratios using one single CNN instead of multiple CNNs corresponding to multiple ratios. In response, we propose compression-aware joint-training and switchable batch normalization. We use ResNet18, ResNet50, and MobileNetV2 on ImageNet to demonstrate our design, achieve inference-time compression ratio adjustability, and reduce external memory access bandwidth requirements. The result shows that our proposed strategies can maintain the Top-1 accuracy and reduce external memory access by at most 22.7x similar to 28.3x only using a single CNN model with sets of BN parameters corresponding to multiple compression ratios.