基于特征符号化和Transformer的无参考图像质量评价方法  

No-reference image quality assessment based on feature tokenizer and Transformer

作　　者：宋巍 李嘉瑾 刘晓晨 刘智翔 石少华[2] SONG Wei;LI Jia-jin;LIU Xiao-chen;LIU Zhi-xiang;SHI Shao-hua(College of Information,Shanghai Ocean University,Shanghai 201306,China;East China Sea Survey Center,State Oceanic Administration,Shanghai 200137,China)

机构地区：[1]上海海洋大学信息学院,上海201306 [2]国家海洋局东海勘察中心,上海200137

出　　处：《液晶与显示》2023年第3期356-367,共12页Chinese Journal of Liquid Crystals and Displays

基　　金：国家自然科学基金(No.61972240)。

摘　　要：基于深度学习的无参考图像质量评价方法目前存在语义关联性不足或模型训练要求高的问题,为此,本文提出了一种基于语义特征符号化和Transformer的无参考图像质量评价方法。首先使用深层卷积神经网络提取图像的高层语义特征;然后将语义特征映射成视觉特征符号,并基于Transformer自注意力机制对视觉特征符号之间的关系进行建模,提取图像的全局特征,同时使用浅层神经网络提取底层局部图像特征,捕捉图像低级失真信息;最后结合全局图像信息与局部图像信息,对图像质量进行预测。为了验证模型的精度和鲁棒性,以相关系数PLCC和SROCC作为评价指标,在5个主流的图像质量评价数据集和1个水下图像质量评价数据集上进行了实验,并将本文提出的方法与15种传统和基于深度学习的无参考图像质量评价方法进行了对比。实验结果表明,本文方法以较少的参数量(大约1.56 MB)在各类数据集上均取得了优越的性能,尤其在多重失真数据集LIVE-MD上将SROCC提升到了0.958,证明在复杂的失真情况下仍能准确评估图像质量,本文网络结构能满足实际应用场景。The no-reference IQA methods based on deep learning have problems of insufficient semantic relevance or high model training requirements.This paper proposes a no-reference IQA based on semantic visual feature tokens and Transformer(VTT-IQA).We firstly use a deep convolutional neural network to extract high-level semantic features of the image,and then map the semantic features to visual feature tokens.Subsequently,the relationship between visual feature tokens is modelled based on the Transformer self-attention mechanism to extract the global information.Meanwhile,a shallow neural network is used to extract the low-level local features of the image and capture its distortion information.Finally,the highlevel semantic information and the low-level visual information are integrated to accurately predict the image quality.In order to verify the superiority and robustness of our proposed model,we compared our method with 15 traditional and deep learning based non-reference IQA methods on five mainstream IQA datasets and one underwater IQA dataset,using PLCC and SROCC as the performance evaluation metrics.The experimental results show that the proposed method achieves superior performance with less parameters(about 1.56 MB).Especially,VTT-IQA achieves 0.958 of SROCC on LIVE-MD that contains multiply distorted images.It is proved that VTT-IQA can still accurately evaluate the image quality under complex distortion,and can meet the practical application.

关 键 词：图像质量 无参考图像质量评价 TRANSFORMER 自注意力 特征符号 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]