基于半监督学习的长尾时序动作检测  

Long-tailed Temporal Action Detection Based on Semi-supervised Learning

作　　者：王雨虹 武港山 王利民 WANG Yu-Hong;WU Gang-Shan;WANG Li-Min(State Key Laboratory for Novel Software Technology(Nanjing University),Nanjing 210023,China)

机构地区：[1]计算机软件新技术国家重点实验室(南京大学),江苏南京210023

出　　处：《软件学报》2025年第2期625-643,共19页Journal of Software

基　　金：科技创新2030—“新一代人工智能”重大项目(2022ZD0160900);国家自然科学基金(62076119,61921006)。

摘　　要：现实世界中的数据标签分布往往呈现长尾效应,即少部分类别占据绝大多数样本,时序动作检测问题也不例外.现有的时序动作检测方法往往缺乏对少样本类别的关注,即充分建模样本数量多的头部类别,而忽视了样本数量少的尾部类别.对长尾时序动作检测问题进行了系统的定义,并针对长尾时序动作检测问题,提出一种基于半监督学习的加权类别重平衡自训练方法,充分利用现实世界中存在的大规模无标签数据,来重平衡训练样本中的标签分布,改善模型对尾部类别的拟合效果.还针对时序动作检测任务,提出一种伪标签损失加权方法,使模型训练更加稳定.在THUMOS14和HACS Segments数据集上进行实验,并分别利用THUMOS15数据集和ActivityNet1.3数据集中的视频样本来构成相应的无标签数据集.此外,还针对视频审核应用需求,收集Dance数据集,包括35个动作类别、6632个有标签视频和13264个无标签视频,并保留数据分布显著的长尾效应.使用多种基线模型,在THUMOS14、HACS Segments和Dance数据集上进行实验.实验结果表明,所提出的加权类别重平衡自训练方法可以提高模型对尾部动作类别的检测效果,并且能应用于不同的基线时序动作检测模型提升其性能.The label distribution in the real world often shows the long-tail effect,where a small number of categories account for the vast majority of samples.The temporal action detection problem is no exception.The existing temporal action detection methods often focus on the head categories with a large number of samples,while neglecting the few-sample categories.This study systematically defines the longtail temporal action detection problem and proposes a weighted class-rebalancing self-training method(WCReST)based on a semisupervised learning framework.WCReST makes full use of the large-scale unlabeled data that exists in the real world to rebalance the label distribution in the training samples to improve the model’s fit for the tail categories.Additionally,a pseudo-label loss weighting method is proposed for the temporal action detection task to enhance the stability of model training.Experiments are conducted on the THUMOS14 and HACS Segments datasets,using video samples from the THUMOS15 and ActivityNet1.3 datasets to form corresponding unlabeled datasets.In addition,the Dance dataset is collected to meet the application requirements of video review,which includes 35 action categories,6632 labeled videos,and 13264 unlabeled videos,preserving the significant long-tail effect in data distribution.A variety of baseline models are used to conduct experiments on the THUMOS14,HACS Segments,and Dance datasets.The results demonstrate that the proposed WCReST can improve the model’s detection performance on tail action categories and can be applied to different baseline temporal action detection models to enhance their performance.

关 键 词：视频分析 时序动作检测 深度长尾学习 半监督学习 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]