An audio-visual speech recognition with a new mandarin audio-visual database

Automatic speech recognition(ASR) by machine has been a goal and an attractive research area for past several decades. In recent years, there has been growing attractive research topic for overcoming certain audio-only recognition problems. Motivated by the multimodal nature of speech, the visual feature is considered to bring in information that dose not existing in the acoustic signal and enables improved system performance over audio-only methods. We first introduce the method for the extraction for the visual feature of the lip. In this paper, we compare four different weighting functions in weighted KNN-based classifiers to recognize ten digits, including 0 to 9, from Mandarin audiovisual speech. The classifiers studied include traditional KNN, weighted KNN, and weighted D-KNN. We also create a new audio-visual database in English and Mandarin. We will describe this audio-visual database and test this database for our proposed system, with some experimental results.