融合生物力学约束与多模态数据的手部重建  

Hand reconstruction incorporating biomechanical constraints and multi-modal data

作　　者：薛皓玮 王美丽[1,2,3] XUE Hao-wei;WANG Mei-li(College of Information Engineering,Northwest A&F University,Xianyang Shaanxi 712100,China;Key Laboratory of Agricultural Internet of Things,Ministry of Agriculture and Rural Affairs,Xianyang Shaanxi 712100,China;Shaanxi Key Laboratory of Agricultural Information Perception and Intelligence Service,Xianyang Shaanxi 712100,China)

机构地区：[1]西北农林科技大学信息工程学院,陕西咸阳712100 [2]农业农村部农业物联网重点实验室,陕西咸阳712100 [3]陕西省农业信息感知与智能服务重点实验室,陕西咸阳712100

出　　处：《图学学报》2023年第4期794-800,共7页Journal of Graphics

基　　金：陕西省林业科学院2021年科技创新计划专项(SXLK2021-0214);陕西省重点研发项目(2022QFY11-03);农村农业部农业物联网重点实验室项目(2018AIOT-09)。

摘　　要：针对目前单目手部重建设备成本高、响应速度迟缓等问题,提出一种利用单目摄像机获取手部形状和姿态进行手部三维重建的方法。该方法采用基于深度学习的架构,使用带有2D和3D注释的图像数据和手部动作捕捉数据进行训练。首先,通过一个关节检测模块(3DHandNet)和一个逆运动学模块(IRNet)准确地回归3D关节位置并将其映射到关节旋转中。其次,通过引入生物力学约束以达到高质量的网格图像对齐以提供实时预测。最后,将得到的预测向量与关节旋转表示输入到手部网格模板来拟合手部形状。仅与回归3D关节位置的方法相比其更适用于计算机视觉和图形学领域的应用。在基准数据集上的实验结果表明,该方法达到了实时的运行性能(60 fps)和较高的重建精度,在姿态估计精度和手部图像对齐方面均优于目前的方法。To address the high cost and slow response of current monocular hand reconstruction,a method for hand 3D reconstruction using a monocular camera to acquire hand shape and posture was proposed.The method adopted a deep learning-based architecture that used image data with 2D and 3D annotations,as well as hand motion capture data for training.Firstly,3D joint positions were accurately regressed and mapped to joint rotations through a joint detection module(3DHandNet)and an inverse kinematic module(IRNet).Then,biomechanical constraints were introduced to achieve high-quality mesh image alignment for real-time predictions.Finally,the resulting prediction vector with joint rotation representation was input to the hand mesh template to fit the hand shape.This approach was more suitable for computer vision and graphics applications compared to only regressing 3D joint positions.Experimental results on a benchmark dataset demonstrated that the proposed method achieves real-time runtime performance(60 fps)and high reconstruction accuracy,outperforming current methods in terms of hand posture estimation accuracy and hand image alignment.

关 键 词：手部三维重建 生物力学约束 多模态数据 手部关节检测模块 逆运动学模块 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]