基于多模态特征融合的人脸物理对抗样本性能预测算法  

Facial Physical Adversarial Example Performance Prediction Algorithm Based on Multi-modal Feature Fusion

作　　者：周风帆 凌贺飞[1] 张锦元 夏紫薇 史宇轩 李平[1] ZHOU Fengfan;LING Hefei;ZHANG Jinyuan;XIA Ziwei;SHI Yuxuan;LI Ping(School of Computer Science and Technology,Huazhong University of Science and Technology,Wuhan 430074,China;Software Development Center,Industrial and Commercial Bank of China,Zhuhai,Guangdong 519080,China)

机构地区：[1]华中科技大学计算机科学与技术学院,武汉430074 [2]中国工商银行软件开发中心,广东珠海519080

出　　处：《计算机科学》2023年第8期280-285,共6页Computer Science

基　　金：国家自然科学基金(61972169);国家重点研发计划(2019QY(Y)0202,2022YFB2601802);湖北省重点研发计划(2022BAA046,2022BAA042);武汉基础研究知识创新项目(2020010601012182);中国博士后科学基金(2022M711251)。

摘　　要：人脸物理对抗样本攻击(Facial Physical Adversarial Attack,FPAA)指攻击者通过粘贴或佩戴物理对抗样本,如打印的眼镜、纸片等,在摄像头下被识别成特定目标的人脸,或者让人脸识别系统无法识别的攻击方式。已有FPAA的性能评测会受到多种环境因素的影响,且需要多个人工操作的环节,导致性能评测效率非常低下。为了减少人脸物理对抗样本性能评测方面的工作量,结合数字图片和环境因素之间的多模态性,提出了多模态特征融合预测算法(Multimodal Feature Fusion Prediction Algorithm,MFFP)。具体地,使用不同的网络提取攻击者人脸图片、受害者人脸图片和人脸数字对抗样本图片的特征,使用环境特征网络来提取环境因素中的特征,然后使用一个多模态特征融合网络对这些特征进行融合,多模态特征融合网络的输出即为所预测的人脸物理对抗样本图片和受害者图片之间的余弦相似度。MFFP算法在未知环境、未知FPAA算法的实验场景下取得了0.003的回归均方误差,其性能优于对比算法,验证了MFFP算法对FPAA性能预测的准确性,可以对FPAA性能进行快速评估,同时大幅降低人工操作的工作量。Facial physical adversarial attack(FPAA)refers to a method that an attacker pasting or wearing physical adversary examples,such as printed glasses,paper,to make the face recognition system to recognize his face as the face of a specific target,or make the face recognition system unable to recognize his face under the camera.The existing performance evaluation process of the FPAA can be affected by multiple environmental factors and require multiple manual operations,resulting in very low efficiency of performance evaluation.In order to reduce the workload of evaluating the performance of facial physical adversarial examples,combined with the multimodality between digital images and environmental factors,a multimodal feature fusion prediction algorithm(MFFP)is proposed.Specifically,different networks are used to extract the features of attacker's face images,victim's face images and facial digital adversarialexample images,and the proposed environmental feature extraction network is used to extract the features of environmental factors.A multimodal feature fusion network is proposed to fuse these features.The output of the multimodal feature fusion network is the cosine similarity performance between the predicted facial physical adversarial example image and the victim image.MFFP algorithm achieves a regression mean square error of 0.003 under the experimental scenario of unknown environment and unknown FPAA,which is better than the performance of the baseline.It verifies the accuracy of MFFP algorithm for predicting of the performance of FPAA.Moreover,it verifies that MFFP can quickly evaluate the performance of FPAA,while greatly reduce the workload of manual operation.

关 键 词：人工智能安全 对抗样本 人脸物理对抗样本攻击 性能预测 多模态特征融合 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]