基于SDBN和BLSTM注意力融合的端到端视听双模态语音识别  

End-to-end audiovisual speech recognition based on attention fusion of SDBN and BLSTM

作　　者：王一鸣 陈恳[1] 萨阿卜杜萨拉木·艾海提拉木 WANG Yiming;CHEN Ken;ABUDUSALAMU Aihaiti(Institute of Communication Technology,Ningbo University,Ningbo 315211,China)

机构地区：[1]宁波大学信息科学与工程学院

出　　处：《电信科学》2019年第12期79-89,共11页Telecommunications Science

基　　金：国家自然科学基金资助项目(No.60972063);宁波市自然科学基金资助项目(No.2014A610065);宁波大学科研基金(理)/学科资助项目(No.XKXL1308)~~

摘　　要：提出一种端到端的视听语音识别算法。在该算法中,通过具有瓶颈结构的深度信念网络(deep belief network,DBN)中引入混合的l1/2范数和l1范数构建一种稀疏DBN(sparse DBN,SDBN)来提取稀疏瓶颈特征,从而实现对数据的特征降维,然后用双向长短期记忆网络(bidirectional long short-term memory,BLSTM)在时序上对特征进行模态处理,之后利用一种注意力机制将经过模态处理的唇部视觉信息和音频听觉信息进行自动对齐、融合,最后将融合的视听觉信息通过一个附加了Softmax层的BLSTM进行分类识别。实验表明,该算法能有效地识别视听觉信息,在同类算法中有很好的识别率和顽健性。An end-to-end audiovisual speech recognition algorithm was proposed. In algorithm, a sparse DBN was constructed by introducing mixed l1/2 norm and l1 norm into Deep Belief Network with bottleneck structure to extract the sparse bottleneck features, so as to reduce the dimension of data features, and then a BLSTM was used to model the feature in time series. Then, a attention mechanism was used to align and fuse the lip visual information and audio auditory information automatically. Finally, the fused audiovisual information was classified and identified by a BLSTM with a Softmax layer attached. Experiments show that the algorithm can effectively identify visual and auditory information, and has good recognition rate and robustness in similar algorithms.

关 键 词：端到端 视听语音识别 稀疏瓶颈特征 注意力机制 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]