Attention-Based Bimodal Neural Network Speech Recognition System on FPGA

To further improve the accuracy of speech recognition technology, a neural network speech recognition system based on a field programmable gate array is designed. Firstly, a neural network audiovisual bimodal speech recognition algorithm based on an attention mechanism is designed. Then, a speech recognition platform based on on-site programmable gate arrays is built. The results showed that the word error rate and the character error rate of this research algorithm were 3.17% and 1.56%, respectively, which were significantly lower than the traditional Lip-Reading Network algorithm's 26.24% and 12.56%. The algorithm converged quickly when the training rounds were less than 10 and tended to stabilize when it was 20. The proposed speech recognition platform used many DSP units in its design, with a utilization rate of 83.2%, the lowest power consumption of 2.21W, the highest energy efficiency ratio of 26.15, and the shortest processing time and faster running speed. In summary, the research algorithm can reasonably allocate learning weights, improve training speed, and has certain feasibility and effectiveness because of introducing attention mechanism. It has good application effects in speech recognition, which helps to improve the accuracy of language recognition algorithms and promote communication between humans and machines. © 2025 Slovene Society Informatika. All rights reserved.