The dialectal and nonnative accents of speakers are challenge questions when spreading and developing the mandarin speech recognition system. This paper describes an integrated way which combines the rule-based data-driven (DD) method with the experts' knowledge to make acoustic models in automatic speech recognition (ASR). The aim is to get regular pairs of the pronunciation variation by statistics. Then, on the basis of this, we can construct the preliminary scheme of mandarin multi-pronunciation dictionary for minority accent in Yunnan. The combined method consists of the following steps. Firstly, baseline hidden Markov models (HMM) were trained by using the project 863 standard Mandarin corpus. Secondly, the nonnative speech data from Dai area, Lisu area and Naxi area in Yunnan was transcribed with the baseline HMMs. In addition, the transcribed result was aligned with the reference transcription through dynamic programming. After calculating of the confusion matrix, we analyze the error pairs due to substitute error at the level of base syllables, initials and finals. Next, we consider the regular mandarin pronunciation variation of national language in Yunnan. Many interesting and useful linguistic phenomena which are necessary for the advancement of nonnative Mandarin speech recognition technology were observed in our experiments.
