Artificial multisensory integration nervous system with haptic and iconic perception behaviors

Sensory neurons integrate multiple sensory inputs into a synthetical perception to monitor complex environments, which significantly impact on how we perceive the world. However, a true implementation of such an electronic device to mimic multiple sensory modalities, especially the coordinating behaviors of multisensory integration, has rarely been reported thus far. Herein, an artificial multisensory integration nervous system with haptic and iconic perception behaviors is developed by integrating flexible triboelectric nanogenerator (TENG) with organic photosynaptic transistor, where lead-free perovskite (Cs2AgBiBr6) quantum dots (QDs) are firstly presented. Meanwhile, Inverse effectiveness effect and Temporal congruency, which are the primary principles of multisensory integration in biological perceptual system, are firstly precisely mimicked in our system. Moreover, the pattern recognition results show that multisensory integration exhibited obviously higher accuracy than single sense, which is consistent with biological system. Furthermore, 3 x 3 pixel array of artificial photosynaptic circuits actuated with TENG is designed and demonstrated for the recognition of various objects in extreme environments, which further verifies the superiority of the multisensory integration with improved image contrast, accuracy of subsequent pattern recognition, and environment-adaptable perception behaviors. This work provides new insight into developing artificial multisensory integration nerve in neuromorphic perceptions and neuromorphic computing, which is of great significance in future human-machine interaction, sophisticated robotic prostheses and neurorobotic system.