多模式汉语连续语音识别中视觉特征的提取和应用  

To Extract and Apply Visual Features in Mandarin Multimodal Continuous Speech Recognition

作　　者：刘鹏[1] 王作英[1] 

机构地区：[1]清华大学电子工程系网络与人机语音通信研究所,北京100084

出　　处：《中文信息学报》2004年第4期79-84,共6页Journal of Chinese Information Processing

基　　金：国家 8 6 3计划资助项目 (2 0 0 1AA1 1 4 0 71 )

摘　　要：本文对在汉语多模式汉语语音识别系统中利用视觉特征进行了研究 ,给出了基于多流隐马尔科夫模型 (Multi streamHMM ,MSHMM)的听视觉融合方案 ,并对有关视觉特征的两项关键技术 :嘴唇定位和视觉特征提取进行了详细讨论。首先 ,我们研究了基于模板匹配的嘴唇跟踪方法 ;然后研究了基于线性变换的低级视觉特征 ,并与基于动态形状模型的特征作了比较 ;实验结果表明 ,引入视觉信息后无噪环境下语音识别声学层首选错误率相对下降 36 0 9% 。In this paper, we investigate on the using of visual feature in Mandarin multimodal speech recognition. The audio visual fusion strategy based on multi stream hidden Markov model is presented. Then key technologies about visual feature, including lip location and visual feature extraction, are discussed. Firstly, we research on the lip location algorithm based on model matching and the low Subsequently, the low level visual feature based on linear transform is investigated and compared to the high level visual feature based on active shape models. It is shown by experiments that the word error rate of the first candidate of acoustic level is reduced by 36 09% relatively with visual feature used, compared to audio speech recognition system. It is also demonstrated from more experiments that our audio visual system provides significant robustness enhancement in noise environment.

关 键 词：计算机应用 中文信息处理 多模式 听一视觉融合 视觉特征提取 鲁棒性 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]