基于异质特征解构的多模态识别方法  

Multimodal recognition method based on heterogeneous feature deconstruction

作　　者：刘伯文[1] 田兆楠 齐跃 韩光照 王兴梅[2] LIU Bowen;TIAN Zhaonan;QI Yue;HAN Guangzhao;WANG Xingmei(China National Aeronautical Radio Electronics Research Institute,Shanghai 200241,China;College of Computer Science and Technology,Harbin Engineering University,Harbin 150001,China;NO.703 Research Institute Of China State Shipbuilding Company Limited,Harbin 150078,China;Harbin Electric Corporation Marine Intelligent Equipment Company Limited,Harbin 150028,China)

机构地区：[1]中国航空无线电电子研究所,上海200241 [2]哈尔滨工程大学计算机科学与技术学院,黑龙江哈尔滨150001 [3]中国船舶集团有限公司第七〇三研究所,黑龙江哈尔滨150078 [4]哈尔滨电气集团海洋智能装备有限公司,黑龙江哈尔滨150028

出　　处：《应用科技》2024年第3期161-168,共8页Applied Science and Technology

基　　金：国家级重点实验室开放基金项目(KY10600220048).

摘　　要：为了解决多模态识别模型因异构模态数据分布之间存在交叉重叠,造成在提取异质特征过程中容易出现特征冗余的问题,提出基于异质特征解构(heterogeneous feature deconstruction,HFD)的多模态识别方法,即构建异质特征解构模型,通过梯度下降的方式训练特性特征提取器,并以梯度反转的方式训练共性特征提取器,提取具有不同模态特质的模态特性特征,以及具有模态不变属性的模态共性特征,进一步利用共性特征增强损失,提高共性特征间的相似度,解决异质特征之间冗余度高的问题。在CMU-MOSEI数据集上的对比实验和消融实验结果验证了基于异质特征解构的多模态识别方法能够有效提升识别性能。In order to solve the problem of feature redundancy in the process of extracting heterogeneous features due to the difference between the distribution of heterogeneous modal data,this paper proposes a multi-modal recognition method based on the deconstruction of heterogeneous features.That is,build a heterogeneous feature deconstruction(HFD)model,train the feature extractor through gradient descent,and train the common feature extractor through gradient inversion to extract modal characteristic features with different modal characteristics.And modal common features with modal invariable properties,further use the common features to enhance the loss,improve the similarity between common features,and solve the problem of high redundancy between heterogeneous features.The results of comparison and ablation experiments on the CMU-MOSEI dataset verify that the proposed multi-modal recognition method based on heterogeneous feature deconstruction can effectively improve the recognition performance.

关 键 词：多模态融合 异质特征 特征提取 梯度反转 余弦相似度 情感识别 特征解构 模态不变空间 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]