基于单流网络的语音-人脸的跨模态学习方法  

Cross Modal Learning Method of Speech Face via Single Stream Network

作　　者：钟方昊 卜凡亮[1] 秦昊铭 ZHONG Fang-hao;BU Fan-liang;QIN Hao-ming(School of Information Network Security,Peoples Public Security University of China,Beijing 100038,China)

机构地区：[1]中国人民公安大学信息网络安全学院,北京100038

出　　处：《科学技术与工程》2025年第11期4638-4646,共9页Science Technology and Engineering

基　　金：中国人民公安大学安全防范工程双一流创新研究专项(2023SYL08)。

摘　　要：现有的语音-人脸跨模态关联学习方法多采用双流网络结构,在降低计算复杂度、模型轻量化和高效特征融合方面还面临一些挑战,为了改善模型性能,提高跨模态学习的效率,提出一种基于单流网络的语音-人脸的跨模态学习方法。首先,将预处理的两种模态数据送入单流特征提取网络,利用基于类信息的损失函数学习提取两种模态的有效特征,接着对提取的两种模态特征向量进行基于注意力机制的特征融合,最后使用余弦相似度算法和交叉熵损失相结合的方法来学习两种模态的关联,从而完成跨模态关联学习任务。实验结果表明,本文提出的方法在语音-人脸跨模态验证、匹配和检索任务上均取得了良好的效果,在考虑网络结构轻量化和灵活性的同时保证了优秀的性能。Existing methods for audio-visual cross-modal association learning often adopt a dual-stream network structure,but they still face challenges in reducing computational complexity,model light weighting,and efficient feature fusion.To improve model performance and enhance the efficiency of cross-modal learning,a single-stream network-based approach for audio-visual cross-modal learning was proposed.Firstly,preprocessed data from both modalities were fed into a single-stream feature extraction network,where a class-information-based loss function was employed to learn and extract feature vectors from both modalities.Subsequently,attention-based feature fusion was performed on the extracted feature vectors from both modalities.Finally,a combination of cosine similarity algorithm and cross-entropy loss was used to learn the association between the two modalities,thus completing the cross-modal association learning task.Experimental results demonstrate that the proposed method achieves promising performance in audio-visual cross-modal verification,matching,and retrieval tasks,ensuring excellent performance while considering the lightness and flexibility of the network structure.

关 键 词：关联学习 语音-人脸跨模态 单流网络 特征融合 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]