Action Recognition From a Single Coded Image

被引:5
|
作者
Kumawat, Sudhakar [1 ]
Okawara, Tadashi [2 ]
Yoshida, Michitaka [2 ]
Nagahara, Hajime [1 ]
Yagi, Yasushi [3 ]
机构
[1] Osaka Univ, Inst Databil Sci, Suita, Osaka 5650871, Japan
[2] Osaka Univ, Grad Sch Informat Sci & Technol, Suita, Osaka 5650871, Japan
[3] Osaka Univ, Inst Sci & Ind Res, Suita, Osaka 5650871, Japan
关键词
Image recognition; Videos; Image reconstruction; Cameras; Sensors; Task analysis; Computational modeling; Action recognition; coded exposure image; computational photography; knowledge distillation;
D O I
10.1109/TPAMI.2022.3196350
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
The unprecedented success of deep convolutional neural networks (CNN) on the task of video-based human action recognition assumes the availability of good resolution videos and resources to develop and deploy complex models. Unfortunately, certain budgetary and environmental constraints on the camera system and the recognition model may not be able to accommodate these assumptions and require reducing their complexity. To alleviate these issues, we introduce a deep sensing solution to directly recognize human actions from coded exposure images. Our deep sensing solution consists of a binary CNN-based encoder network that emulates the capturing of a coded exposure image of a dynamic scene using a coded exposure camera, followed by a 2D CNN for recognizing human action in the captured coded exposure image. Furthermore, we propose a novel knowledge distillation framework to jointly train the encoder and the action recognition model and show that the proposed training approach improves the action recognition accuracy by an absolute margin of 6.2%, 2.9%, and 7.9% on Something(2)-v2, Kinetics-400, and UCF-101 datasets, respectively, in comparison to our previous approach. Finally, we built a prototype coded exposure camera using LCoS to validate the feasibility of our deep sensing solution. Our evaluation of the prototype camera show results that are consistent with the simulation results.
引用
收藏
页码:4109 / 4121
页数:13
相关论文
共 50 条
  • [11] Action Recognition with Dynamic Image Networks
    Bilen, Hakan
    Fernando, Basura
    Gavves, Efstratios
    Vedaldi, Andrea
    IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2018, 40 (12) : 2799 - 2813
  • [12] Investigating image stitching for action recognition
    Li, Yufeng
    Li, Ping'an
    Lei, Daozhong
    Shi, Yingchun
    Tan, Lixin
    MULTIMEDIA TOOLS AND APPLICATIONS, 2018, 77 (03) : 3279 - 3286
  • [13] Dynamic Image Networks for Action Recognition
    Bilen, Hakan
    Fernando, Basura
    Gavves, Efstratios
    Vedaldi, Andrea
    Gould, Stephen
    2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, : 3034 - 3042
  • [14] Investigating image stitching for action recognition
    Yufeng Li
    Ping’an Li
    Daozhong Lei
    Yingchun Shi
    Lixin Tan
    Multimedia Tools and Applications, 2018, 77 : 3279 - 3286
  • [15] Action Recognition from Single Timestamp Supervision in Untrimmed Videos
    Moltisanti, Davide
    Fidler, Sanja
    Damen, Dima
    2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019), 2019, : 9907 - 9916
  • [16] Human Action Recognition Algorithm Based on Improved ResNet and Skeletal Keypoints in Single Image
    Lin, Yixue
    Chi, Wanda
    Sun, Wenxue
    Liu, Shicai
    Fan, Di
    MATHEMATICAL PROBLEMS IN ENGINEERING, 2020, 2020
  • [17] Action recognition from mutually incoherent pose bases in static image
    Qian, Yinzhong
    Chen, Wenbin
    Shen, I-fan
    IET COMPUTER VISION, 2018, 12 (03) : 233 - 240
  • [18] Automated Recognition of a Wall between Windows from a Single Image
    Zhang, Yaowen
    Huo, Linsheng
    Li, Hongnan
    JOURNAL OF SENSORS, 2017, 2017
  • [19] 3-D OBJECT RECOGNITION FROM A SINGLE IMAGE
    COHEN, FS
    CAYULA, JFP
    PROCEEDINGS OF THE SOCIETY OF PHOTO-OPTICAL INSTRUMENTATION ENGINEERS, 1985, 521 : 7 - 15
  • [20] Dominant Plane Recognition in Interior Scenes from a Single Image
    de Jesus Osuna-Coutino, J. A.
    Martinez-Carranza, Jose
    Arias-Estrada, Miguel
    Mayol-Cuevas, Walterio
    2016 23RD INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), 2016, : 1923 - 1928