Lightweight Classroom Student Action Recognition Method Based on Spatiotemporal Multimodal Feature Fusion  

作　　者：Shaodong Zou Di Wu Jianhou Gan Juxiang Zhou Jiatian Mei 

机构地区：[1]Key Laboratory of Education Informatization for Nationalities,Ministry of Education,Yunnan Normal University,Kunming,650500,China [2]Yunnan Key Laboratory of Smart Education,Yunnan Normal University,Kunming,650500,China

出　　处：《Computers, Materials & Continua》2025年第4期1101-1116,共16页计算机、材料和连续体(英文)

基　　金：supported by the National Natural Science Foundation of China under Grant 62107034;the Major Science and Technology Project of Yunnan Province(202402AD080002);Yunnan International Joint R&D Center of China-Laos-Thailand Educational Digitalization(202203AP140006).

摘　　要：The task of student action recognition in the classroom is to precisely capture and analyze the actions of students in classroom videos,providing a foundation for realizing intelligent and accurate teaching.However,the complex nature of the classroom environment has added challenges and difficulties in the process of student action recognition.In this research article,with regard to the circumstances where students are prone to be occluded and classroom computing resources are restricted in real classroom scenarios,a lightweight multi-modal fusion action recognition approach is put forward.This proposed method is capable of enhancing the accuracy of student action recognition while concurrently diminishing the number of parameters of the model and the Computation Amount,thereby achieving a more efficient and accurate recognition performance.In the feature extraction stage,this method fuses the keypoint heatmap with the RGB(Red-Green-Blue color model)image.In order to fully utilize the unique information of different modalities for feature complementarity,a Feature Fusion Module(FFE)is introduced.The FFE encodes and fuses the unique features of the two modalities during the feature extraction process.This fusion strategy not only achieves fusion and complementarity between modalities,but also improves the overall model performance.Furthermore,to reduce the computational load and parameter scale of the model,we use keypoint information to crop RGB images.At the same time,the first three networks of the lightweight feature extraction network X3D are used to extract dual-branch features.These methods significantly reduce the computational load and parameter scale.The number of parameters of the model is 1.40 million,and the computation amount is 5.04 billion floating-point operations per second(GFLOPs),achieving an efficient lightweight design.In the Student Classroom Action Dataset(SCAD),the accuracy of the model is 88.36%.In NTU 60(Nanyang Technological University Red-Green-Blue-Depth RGB+Ddataset with 60 categorie

关 键 词：Action recognition student classroom action multimodal fusion lightweight model design 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]