Overcoming Oscillations in Quantization-Aware Training

被引:0
|
作者
Nagel, Markus [1 ]
Fournarakis, Marios [1 ]
Bondarenko, Yelysei [1 ]
Blankevoort, Tijmen [1 ]
机构
[1] Qualcomm AI Res, San Diego, CA 92121 USA
关键词
D O I
暂无
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
When training neural networks with simulated quantization, we observe that quantized weights can, rather unexpectedly, oscillate between two grid-points. The importance of this effect and its impact on quantization-aware training (QAT) are not well-understood or investigated in literature. In this paper, we delve deeper into the phenomenon of weight oscillations and show that it can lead to a significant accuracy degradation due to wrongly estimated batch-normalization statistics during inference and increased noise during training. These effects are particularly pronounced in low-bit (<= 4-bits) quantization of efficient networks with depth-wise separable layers, such as MobileNets and EfficientNets. In our analysis we investigate several previously proposed QAT algorithms and show that most of these are unable to overcome oscillations. Finally, we propose two novel QAT algorithms to overcome oscillations during training: oscillation dampening and iterative weight freezing. We demonstrate that our algorithms achieve state-of-the-art accuracy for low-bit (3 & 4 bits) weight and activation quantization of efficient architectures, such as MobileNetV2, MobileNetV3, and EfficentNet-lite on ImageNet. Our source code is available at https://github.com/qualcomm-ai- research/oscillations-qat.
引用
收藏
页数:13
相关论文
共 50 条
  • [1] Overcoming Forgetting Catastrophe in Quantization-Aware Training
    Chen, Ting-An
    Yang, De-Nian
    Chen, Ming-Syan
    [J]. 2023 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2023), 2023, : 17312 - 17321
  • [2] AdaQAT: Adaptive Bit-Width Quantization-Aware Training
    Gernigon, Cedric
    Filip, Silviu-Ioan
    Sentieys, Olivier
    Coggiola, Clement
    Bruno, Mickael
    [J]. 2024 IEEE 6TH INTERNATIONAL CONFERENCE ON AI CIRCUITS AND SYSTEMS, AICAS 2024, 2024, : 442 - 446
  • [3] A Robust, Quantization-Aware Training Method for Photonic Neural Networks
    Oikonomou, A.
    Kirtas, M.
    Passalis, N.
    Mourgias-Alexandris, G.
    Moralis-Pegios, M.
    Pleros, N.
    Tefas, A.
    [J]. ENGINEERING APPLICATIONS OF NEURAL NETWORKS, EAAAI/EANN 2022, 2022, 1600 : 427 - 438
  • [4] Disentangled Loss for Low-Bit Quantization-Aware Training
    Allenet, Thibault
    Briand, David
    Bichler, Olivier
    Sentieys, Olivier
    [J]. 2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS, CVPRW 2022, 2022, : 2787 - 2791
  • [5] Approximation- and Quantization-Aware Training for Graph Neural Networks
    Novkin, Rodion
    Klemme, Florian
    Amrouch, Hussam
    [J]. IEEE TRANSACTIONS ON COMPUTERS, 2024, 73 (02) : 599 - 612
  • [6] Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training
    Sakr, Charbel
    Dai, Steve
    Venkatesan, Rangharajan
    Zimmer, Brian
    Dally, William J.
    Khailany, Brucek
    [J]. INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 162, 2022, : 19123 - 19138
  • [7] Quantization-aware phase retrieval
    Mukherjee, Subhadip
    Seelamantula, Chandra Sekhar
    [J]. INTERNATIONAL JOURNAL OF WAVELETS MULTIRESOLUTION AND INFORMATION PROCESSING, 2022, 20 (03)
  • [8] Quantization-aware training for low precision photonic neural networks
    Kirtas, M.
    Oikonomou, A.
    Passalis, N.
    Mourgias-Alexandris, G.
    Moralis-Pegios, M.
    Pleros, N.
    Tefas, A.
    [J]. NEURAL NETWORKS, 2022, 155 : 561 - 573
  • [9] Low Precision Quantization-aware Training in Spiking Neural Networks with Differentiable Quantization Function
    Shymyrbay, Ayan
    Fouda, Mohammed E.
    Eltawil, Ahmed
    [J]. 2023 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS, IJCNN, 2023,
  • [10] SQUAT: Stateful Quantization-Aware Training in Recurrent Spiking Neural Networks
    Venkatesh, Sreyes
    Marinescu, Razvan
    Eshraghian, Jason K.
    [J]. 2024 NEURO INSPIRED COMPUTATIONAL ELEMENTS CONFERENCE, NICE, 2024,