Hierarchical Vector-Quantized Variational Autoencoder and Vector Credibility Mechanism for High-Quality Image Inpainting

被引:0
|
作者
Li, Cheng [1 ]
Xu, Dan [1 ]
Chen, Kuai [2 ]
机构
[1] Yunnan Univ, Sch Informat Sci & Engn, Kunming 650106, Peoples R China
[2] Yunnan Univ, Sch Govt, Kunming 650106, Peoples R China
基金
中国国家自然科学基金;
关键词
image inpainting; VQ-VAE; vector credibility; codebook;
D O I
10.3390/electronics13101852
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Image inpainting infers the missing areas of a corrupted image according to the information of the undamaged part. Many existing image inpainting methods can generate plausible inpainted results from damaged images with the fast-developed deep-learning technology. However, they still suffer from over-smoothed textures or textural distortion in the cases of complex textural details or large damaged areas. To restore textures at a fine-grained level, we propose an image inpainting method based on a hierarchical VQ-VAE with a vector credibility mechanism. It first trains the hierarchical VQ-VAE with ground truth images to update two codebooks and to obtain two corresponding vector collections containing information on ground truth images. The two vector collections are fed to a decoder to generate the corresponding high-fidelity outputs. An encoder then is trained with the corresponding damaged image. It generates vector collections approximating the ground truth by the help of the prior knowledge provided by the codebooks. After that, the two vector collections pass through the decoder from the hierarchical VQ-VAE to produce the inpainted results. In addition, we apply a vector credibility mechanism to promote vector collections from damaged images and approximate vector collections from ground truth images. To further improve the inpainting result, we apply a refinement network, which uses residual blocks with different dilation rates to acquire both global information and local textural details. Extensive experiments conducted on several datasets demonstrate that our method outperforms the state-of-the-art ones.
引用
收藏
页数:17
相关论文
共 50 条
  • [1] Vector-Quantized Variational AutoEncoder for pansharpening
    Talbi, Farid
    Elmezouar, Miloud Chikr
    Boutellaa, Elhocine
    Alim, Fatiha
    INTERNATIONAL JOURNAL OF REMOTE SENSING, 2023, 44 (20) : 6329 - 6349
  • [2] Vector-quantized Variational Autoencoder for Phase-aware Speech Enhancement
    Tuan Vu Ho
    Quoc Huy Nguyen
    Akagi, Masato
    Unoki, Masashi
    INTERSPEECH 2022, 2022, : 176 - 180
  • [3] Leveraging Vector-Quantized Variational Autoencoder Inner Metrics for Anomaly Detection
    Gangloff, Hugo
    Pham, Minh-Tan
    Courtrai, Luc
    Lefevre, Sebastien
    2022 26TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), 2022, : 435 - 441
  • [4] Vector-Quantized Autoencoder With Copula for Collaborative Filtering
    Wang, Guanyu
    Zhong, Ting
    Xu, Xovee
    Zhang, Kunpeng
    Zhou, Fan
    Wang, Yong
    PROCEEDINGS OF THE 30TH ACM INTERNATIONAL CONFERENCE ON INFORMATION & KNOWLEDGE MANAGEMENT, CIKM 2021, 2021, : 3458 - 3462
  • [5] CRANK: AN OPEN-SOURCE SOFTWARE FOR NONPARALLEL VOICE CONVERSION BASED ON VECTOR-QUANTIZED VARIATIONAL AUTOENCODER
    Kobayashi, Kazuhiro
    Huang, Wen-Chin
    Wu, Yi-Chiao
    Tobing, Patrick Lumban
    Hayashi, Tomoki
    Toda, Tomoki
    2021 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP 2021), 2021, : 5934 - 5938
  • [6] Bone-conducted Speech Enhancement Using Vector-quantized Variational Autoencoder and Gammachirp Filterbank Cepstral Coefficients
    Quoc-Huy Nguyen
    Unoki, Masashi
    2022 30TH EUROPEAN SIGNAL PROCESSING CONFERENCE (EUSIPCO 2022), 2022, : 21 - 25
  • [7] WEAKLY SUPERVISED MARINE ANIMAL DETECTION FROM REMOTE SENSING IMAGES USING VECTOR-QUANTIZED VARIATIONAL AUTOENCODER
    Pham, Minh-Tan
    Gangloff, Hugo
    Lefevre, Sebastien
    IGARSS 2023 - 2023 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM, 2023, : 5559 - 5562
  • [8] VQCNIR: Clearer Night Image Restoration with Vector-Quantized Codebook
    Zou, Wenbin
    Gao, Hongxia
    Ye, Tian
    Chen, Liang
    Yang, Weipeng
    Huang, Shasha
    Chen, Hongshen
    Chen, Sixiang
    THIRTY-EIGHTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, VOL 38 NO 7, 2024, : 7873 - 7881
  • [9] Predictive Vector Quantized Variational AutoEncoder for Spectral Envelope Quantization
    Srikotr, Tanasan
    Mano, Kazunori
    2020 INTERNATIONAL CONFERENCE ON ELECTRONICS, INFORMATION, AND COMMUNICATION (ICEIC), 2020,
  • [10] LEARNING PRODUCT CODEBOOKS USING VECTOR-QUANTIZED AUTOENCODERS FOR IMAGE RETRIEVAL
    Wu, Hanwei
    Flierl, Markus
    2019 7TH IEEE GLOBAL CONFERENCE ON SIGNAL AND INFORMATION PROCESSING (IEEE GLOBALSIP), 2019,