A Robust Framework for Multimodal Sentiment Analysis with Noisy Labels Generated from Distributed Data Annotation  

作　　者：Kai Jiang Bin Cao Jing Fan 

机构地区：[1]School of Computer Science and Technology,Zhejiang University of Technology,Hangzhou,310023,China

出　　处：《Computer Modeling in Engineering & Sciences》2024年第6期2965-2984,共20页工程与科学中的计算机建模（英文）

基　　金：supported by STI 2030-Major Projects 2021ZD0200400;National Natural Science Foundation of China(62276233 and 62072405);Key Research Project of Zhejiang Province(2023C01048).

摘　　要：Multimodal sentiment analysis utilizes multimodal data such as text,facial expressions and voice to detect people’s attitudes.With the advent of distributed data collection and annotation,we can easily obtain and share such multimodal data.However,due to professional discrepancies among annotators and lax quality control,noisy labels might be introduced.Recent research suggests that deep neural networks(DNNs)will overfit noisy labels,leading to the poor performance of the DNNs.To address this challenging problem,we present a Multimodal Robust Meta Learning framework(MRML)for multimodal sentiment analysis to resist noisy labels and correlate distinct modalities simultaneously.Specifically,we propose a two-layer fusion net to deeply fuse different modalities and improve the quality of the multimodal data features for label correction and network training.Besides,a multiple meta-learner(label corrector)strategy is proposed to enhance the label correction approach and prevent models from overfitting to noisy labels.We conducted experiments on three popular multimodal datasets to verify the superiority of ourmethod by comparing it with four baselines.

关 键 词：Distributed data collection multimodal sentiment analysis meta learning learn with noisy labels 

分 类 号：TB53[理学—物理] TP3[理学—声学]