基于多模态融合的开放域三维模型检索算法  

Open-set 3D model retrieval algorithm based on multi-modal fusion

作　　者：毛福新 杨旭[1] 程嘉强 彭涛 MAO Fuxin;YANG Xu;CHENG Jiaqiang;PENG Tao(Engineering Training Center,Tianjin University of Technology and Education,Tianjin 300222,China;Tianjin Huada Technology Limited Company,Tianjin 300131,China;College of Automobile and Transportation,Tianjin University of Technology and Education,Tianjin 300222,China)

机构地区：[1]天津职业技术师范大学工程实训中心,天津300222 [2]天津华大科技有限公司,天津300131 [3]天津职业技术师范大学汽车与交通学院,天津300222

出　　处：《浙江大学学报（工学版）》2024年第1期61-70,共10页Journal of Zhejiang University：Engineering Science

基　　金：天津市多元投入基金重点资助项目(21JCZDJC00700)。

摘　　要：为了满足开放域下海量新增模型数据的管理和检索需求,提出开放域三维模型检索算法,可以有效地利用多模态信息的语义一致性.借助无监督算法探寻未知样本间的类别信息,利用该类别信息实现网络模型的参数优化,使得网络模型在开放域条件下具有更好的模型表征性能及检索结果.提出基于Transformer结构的层级化多模态信息融合模型,有效地剔除了多模态间的冗余信息,得到鲁棒性更强的模型表征向量.在数据集ModelNet40上进行实验,通过与其他典型算法的对比实验可知,所提方法在mAP指标上优于所有的对比方法,验证了该方法在检索性能提升上的有效性.An open domain 3D model retrieval algorithm was proposed in order to meet the requirement of management and retrieval of massive new model data under the open domain.The semantic consistency of multimodal information can be effectively used.The category information among unknown samples was explored with the help of unsupervised algorithm.Then the unknown class information was introduced into the parameter optimization process of the network model.The network model has better characterization and retrieval performance in the open domain condition.A hierarchical multi-modal information fusion model based on a Transformer structure was proposed,which could effectively remove the redundant information among the modalities and obtain a more robust model representation vector.Experiments were conducted on the dataset ModelNet40,and the experiments were compared with other typical algorithms.The proposed method outperformed all comparative methods in terms of mAP metrics,which verified the effectiveness of the method in terms of retrieval performance improvement.

关 键 词：机器视觉 多模态融合 开放域检索 三维模型 

分 类 号：TP37[自动化与计算机技术—计算机系统结构]