Preprocessing for Keypoint-Based Sign Language Translation without Glosses

被引:5
|
作者
Kim, Youngmin [1 ]
Baek, Hyeongboo [1 ]
机构
[1] Incheon Natl Univ INU, Dept Comp Sci & Engn, Incheon 22012, South Korea
关键词
computer vision; deep learning; sign language translation; video processing;
D O I
10.3390/s23063231
中图分类号
O65 [分析化学];
学科分类号
070302 ; 081704 ;
摘要
While machine translation for spoken language has advanced significantly, research on sign language translation (SLT) for deaf individuals remains limited. Obtaining annotations, such as gloss, can be expensive and time-consuming. To address these challenges, we propose a new sign language video-processing method for SLT without gloss annotations. Our approach leverages the signer's skeleton points to identify their movements and help build a robust model resilient to background noise. We also introduce a keypoint normalization process that preserves the signer's movements while accounting for variations in body length. Furthermore, we propose a stochastic frame selection technique to prioritize frames to minimize video information loss. Based on the attention-based model, our approach demonstrates effectiveness through quantitative experiments on various metrics using German and Korean sign language datasets without glosses.
引用
收藏
页数:16
相关论文
共 50 条
  • [1] Considerations for meaningful sign language machine translation based on glosses
    Mueller, Mathias
    Jiang, Zifan
    Moryossef, Amit
    Rios, Annette
    Ebling, Sarah
    61ST CONFERENCE OF THE THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, ACL 2023, VOL 2, 2023, : 682 - 693
  • [2] Neural Sign Language Translation Based on Human Keypoint Estimation
    Ko, Sang-Ki
    Kim, Chang Jo
    Jung, Hyedong
    Cho, Choongsang
    APPLIED SCIENCES-BASEL, 2019, 9 (13):
  • [3] Keypoint-Based Keyframe Selection
    Guan, Genliang
    Wang, Zhiyong
    Lu, Shiyang
    Da Deng, Jeremiah
    Feng, David Dagan
    IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 2013, 23 (04) : 729 - 734
  • [4] A rule-based translation from written Spanish to Spanish Sign Language glosses
    Porta, Jordi
    Lopez-Colino, Fernando
    Tejedor, Javier
    Colas, Jose
    COMPUTER SPEECH AND LANGUAGE, 2014, 28 (03): : 788 - 811
  • [5] Data Augmentation for Human Keypoint Estimation Deep Learning based Sign Language Translation
    Park, Chan-Il
    Sohn, Chae-Bong
    ELECTRONICS, 2020, 9 (08) : 1 - 9
  • [6] Neural Machine Translation Methods for Translating Text to Sign Language Glosses
    Zhu, Dele
    Czehmann, Vera
    Avramidis, Eleftherios
    PROCEEDINGS OF THE 61ST ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2023): LONG PAPERS, VOL 1, 2023, : 12523 - 12541
  • [7] DYNAMIC KEYPOINT-BASED ALGORITHM OF OBJECT TRACKING
    Morgacheva, A. I.
    Kulikov, V. A.
    Kosykh, V. P.
    INTERNATIONAL WORKSHOP PHOTOGRAMMETRIC AND COMPUTER VISION TECHNIQUES FOR VIDEO SURVEILLANCE, BIOMETRICS AND BIOMEDICINE, 2017, 42-2 (W4): : 79 - 82
  • [8] A Keypoint-based Fast Object Tracking Algorithm
    Cao, Weihua
    Ling, Qiang
    Li, Feng
    Zheng, Quan
    Wang, Song
    PROCEEDINGS OF THE 35TH CHINESE CONTROL CONFERENCE 2016, 2016, : 4102 - 4106
  • [9] A New Descriptor for Keypoint-Based Background Modeling
    Avola, Danilo
    Bernardi, Marco
    Cascio, Marco
    Cinque, Luigi
    Foresti, Gian Luca
    Massaroni, Cristiano
    IMAGE ANALYSIS AND PROCESSING - ICIAP 2019, PT I, 2019, 11751 : 15 - 25
  • [10] Robust Keypoint Normalization Method for Korean Sign Language Translation using Transformer
    Kim, San
    Kim, Chang Jo
    Park, Han-Mu
    Jeong, Yoonyoung
    Jang, Jin Yea
    Jung, Hyedong
    11TH INTERNATIONAL CONFERENCE ON ICT CONVERGENCE: DATA, NETWORK, AND AI IN THE AGE OF UNTACT (ICTC 2020), 2020, : 1303 - 1305