基于条件生成对抗网络的手语样本骨架缺失关节点修复  

Missing Joint Point Repair of Sign Language Sample Skeleton Based on Conditional Generation Adversarial Networks

作　　者：彭冲 张金艺[1] 楼亮亮[2] Peng Chong;Zhang Jinyi;Lou Liangliang(Key Laboratory of Specialty Fiber Optics and Optical Access Networks,Joint International Research Laboratory of Specialty Fiber Optics and Advanced Communication,Shanghai University,Shanghai 200444;Key Laboratory of Wireless Sensor Network&Communication,Shanghai Institute of Microsystem and Information Technology,Chinese Academy of Sciences,Shanghai 200050)

机构地区：[1]上海大学特种光纤与光接入网重点实验室特种光纤与先进通信国际合作联合实验室,上海200444 [2]中国科学院上海微系统与信息技术研究所无线传感网与通信重点实验室,上海200050

出　　处：《计算机辅助设计与图形学学报》2023年第3期423-433,共11页Journal of Computer-Aided Design & Computer Graphics

基　　金：高等学校学科创新引智计划(111)项目(D20031);上海市教委重点学科资助项目(J50104).

摘　　要：计算机视觉技术由于受到遮挡、视角和光照等因素的影响,对手语样本骨架关节点的检测通常存在缺失,导致手语识别准确率降低.为此,提出基于条件生成对抗网络(CGAN)的手语样本骨架缺失关节点修复方法.首先,通过分析手语样本残缺骨架中关节点的缺失分布情况,构建缺失关节点分布概率模型;其次,对完整骨架引入分布概率模型生成的缺失关节点,将这些残缺骨架用于CGAN框架中生成器和判别器的训练,通过CGAN框架训练好的生成器能够以残缺骨架为条件生成没有缺失的骨架;最后,用生成骨架去补全残缺骨架,即完成了修复.在中国手语数据集CSL上开展实验,生成器迭代训练80次后,生成骨架与完整骨架的平均均方根误差从0.019减小到0.001;在修复骨架缺失关节点的手语样本上,搭建手语识别网络迭代训练120次,与未进行修复相比,其识别准确率从90.6%提升为99.6%.实验结果表明,该方法能够有效地修复缺失关节点,极大地提升手语识别准确率.Due to the influence of occlusion,angle of view,illumination and other factors,the detection of skeleton joint points of sign language samples by computer vision technology is usually missing,which leads to the lower accuracy of sign language recognition.To solve this problem,a new method based on conditional generation adversarial networks(CGAN)was proposed to repair the missing joint points of sign language sample skeleton.Firstly,by analyzing the distribution of missing joint points in the incomplete skeleton of sign language samples,the distribution probability model of missing joint points was constructed.Secondly,the missing joint points generated by the distributed probability model are added into the complete skeleton,and uses these incomplete skeletons for the training of the generator and discriminator in the CGAN framework,the generator trained through the CGAN framework can generate skeletons without missing joint points on the condition of the incomplete skeleton.Finally,we use the generated skeleton to fill the incomplete skeleton and complete the repair.Experiments are carried out on the Chinese sign language dataset CSL,after the generator iteratively trained for 80 times,the average root mean square error between the generated skeleton and the complete skeleton is reduced from 0.019 to 0.001.120 times of iterative training of sign language recognition network was built on the sign language samples with missing joint points repaired,compared with no repair,the recognition accuracy is increased from 90.6%to 99.6%.The experimental results show that the proposed method can effectively repair the missing joints and greatly improve the accuracy of sign language recognition.

关 键 词：手语样本 缺失关节点 条件生成对抗网络 分布概率模型 手语识别 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]