面向多模数据的引导-对齐情绪推理方法  

Guidance⁃alignment emotion inference method for multi⁃modal data

作　　者：张艳[1] 夏雨琪 丁凯 刘阳炀 王年[1] ZHANG Yan;XIA Yuqi;DING Kai;LIU Yangyang;WANG Nian(School of Electronics Information Engineering,Anhui University,Hefei 230601,China)

机构地区：[1]安徽大学电子信息工程学院,合肥230601

出　　处：《东南大学学报(自然科学版)》2025年第2期585-592,共8页Journal of Southeast University:Natural Science Edition

基　　金：国家重点研发计划资助项目(2022YFF0604801);安徽省高校协同创新资助项目(GXXT⁃2022⁃038);合肥市自然科学基金资助项目(202303);皖江中心产业化资助项目(2024340104001649).

摘　　要：表情和声音等微观情绪需近距离交互采集。为了将空间尺度大、数据容易获取的姿态信息作为情绪表达的载体,提出一种基于引导-对齐模块的情绪推理方法。其中引导模块借助面部关键点指导姿态特征的提取,进行帧图像二级筛选;首先提取出同时包含面部关键点和人体姿态的帧图像,通过对每帧图像的欧氏度量筛选保留符合要求的人体姿态帧图像,实现面部特征引导姿态特征的提取;通过特征对数归一化实现姿态对齐模块,姿态特征与面部特征、环境特征共同构成视觉特征,将视觉特征、文本特征和语音特征进行多模态特征融合。实验结果表明,该方法在MEmoR数据集上的Micro⁃F_(1)达到48.86%,一定程度上提升了多模态情绪推理能力。Micro⁃level emotions such as facial expressions and vocal tones require close interaction for collec⁃tion.To utilize posture information,which is easily accessible and spans larger spatial scales,as a vehicle for emotion inference,an emotion inference method based on a guidance⁃alignment module is proposed.The guidance module uses facial keypoints to direct the extraction of posture features and performs a secondary se⁃lection of frame images.First,frames containing both facial keypoints and human posture are extracted.Eu⁃clidean distance is used to filter and retain posture frames that meet the criteria,achieving facial feature⁃guided posture feature extraction.The pose alignment module is implemented through logarithmic normalization of features.The extracted posture features,together with facial features and environmental features,constitute the visual features.The visual features,along with textual and audio features,are integrated for multi⁃modal feature fusion.Experimental results demonstrate that the proposed method achieves a Micro⁃F_(1) score of 48.86%on the MEmoR dataset.The method improves multi⁃modal emotion inference capability to a certain extent.

关 键 词：情绪推理 多模态 特征对数对齐 引导特征 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]