一种多模态隐喻数据集的构建和验证方法  

A Method for Constructing and Validating a Multimodal Metaphor Dataset

作　　者：夏冰[1] 杨瑞楠 董玉 楚世豪 唐崇俊 葛云翔 尹家斌 XIA Bing;YANG Ruinan;DONG Yu;CHU Shihao;TANG Chongjun;GE Yunxiang;YIN Jiabin(Zhongyuan University of Technology,Zhengzhou 450007,China)

机构地区：[1]中原工学院,郑州450007

出　　处：《集成技术》2024年第5期64-73,共10页Journal of Integration Technology

基　　金：河南省科技攻关项目(232102211088);河南省哲学社会科学规划项目(2022BXW018)。

摘　　要：隐喻的目的是启发理解、说服他人。目前,隐喻呈现文本、图像、视频等多模态融合的趋势,因此,识别多模态信息中蕴含的隐喻语义对互联网内容安全具有研究价值。由于缺乏多模态隐喻数据集,难以建立研究模型,因此,当前学者更关注基于文本的隐喻检测。针对这一不足,作者首先从图像-文本、隐喻出现、情感表达和作者意图等角度构建新型多模态隐喻数据集;其次,对数据集的标注者进行Kappa分数计算;最后,借助预训练模型和注意力机制融合图像属性特征、图像实体对象特征和文本特征,构建多模态隐喻检测模型,验证多模态数据集的质量和价值。实验结果表明:具有情感和意图表达的隐喻数据集可提升隐喻模型检测效果,多模态信息间相互关系有助于隐喻的理解。Metaphor has the purpose of inspiring understanding and persuading others.Currently,metaphor presents the trend of multimodal integration of text,images,and videos.Therefore,identifying the metaphorical semantics contained in multimodal contents has research value for Internet content security.Due to the lack of multimodal metaphor datasets,it is difficult to establish research models.Therefore,current scholars pay more attention to text-based metaphor detection.To overcome this shortcoming,the paper first generates a new multimodal metaphor dataset from the perspectives of image-text,metaphor appearance,emotion expression,and author intention.Then,Kappa scores were used to assess the consistency among the annotators of the dataset.Finally,a multimodal metaphor detection model is constructed to verify the quality and value of the multimodal data set by combining image attribute features,image entity features,and text features with the help of a pre-training model and attention mechanism.The experimental results show that the metaphor dataset with emotion and intention can improve the effectiveness of metaphor model detection,and confirm that the interrelationship of multimodal information is helpful for understanding metaphor.

关 键 词：内容安全 多模态隐喻检测 外部知识 多模态数据集 注意力机制 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]