基于HoloLens的增强现实手术导航系统精准融合显示方法  

Accurate Fusion Overlay Method of Augmented Reality Surgical Navigation System Based on HoloLens

作　　者：王君臣[1] 朱辉 胡岩 宋友[2] 陈思颖 Wang Junchen;Zhu Hui;Hu Yan;Song You;Chen Siying(School of Mechanical Engineering and Automation,Beihang University,Beijing 100191,China;School of Software,Beihang University,Beijing 100191,China;Shandong Intelligent Robot Application Technology Research Institute,Jining,Shandong 273500,China;School of Electrical Engineering,University of Jinan,Jinan 250022,China)

机构地区：[1]北京航空航天大学机械工程及自动化学院,北京100191 [2]北京航空航天大学软件学院,北京100191 [3]山东省智能机器人应用技术研究院,山东济宁273500 [4]济南大学自动化与电气工程学院,济南250022

出　　处：《机电工程技术》2023年第6期6-13,18,共9页Mechanical & Electrical Engineering Technology

基　　金：国家重点研发计划(2022YFC2405401)。

摘　　要：传统手术导航系统中医生需要频繁切换视野,存在手眼协同问题,且无法实时获取患者病灶区域的全局视野。近年来Holo-Lens被逐步应用于新兴的增强现实手术导航系统中,但是现有的HoloLens全息空间标定方法多数依赖于用户的主观感受,显示过程中也没有考虑用户行为和周围环境变化带来的误差。对此,提出了一种基于HoloLens的增强现实手术导航系统精准融合显示方法。使用双目相机和视觉标记物完成对现实空间中目标物体的实时追踪。基于手眼标定原理提出了一种稳定的HoloLens离线标定方法,通过多次采集HoloLens全息空间和现实空间中的坐标系转换关系,可以准确求解全息空间与现实空间之间的映射关系。建立融合显示过程的位姿矩阵传递关系后,用户可以快速地进行在线校正,实现虚拟影像与患者病灶区域的精准融合显示。Holo-Lens离线标定过程的平均耗时为4 min,用户在线校正的平均耗时为6.0 s。多个志愿者穿戴HoloLens设备进行融合显示精度验证实验,实验结果表明,校正前HoloLens融合显示的平均角度误差和平均距离误差分别为1.01°和3.95 mm,校正后融合显示的平均距离误差为1.46 mm。所提出的增强现实手术导航系统精准融合显示方法可以准确地标定HoloLens全息空间,经过用户在线校正后可以将虚拟影像与患者病灶区域融合显示,为医生提供准确完整的手术指导信息。实验结果表明该融合显示方法基本满足手术导航系统的精度和时间要求,具有临床应用的潜力。In the traditional surgical navigation system,doctors need to switch the field of view frequently,there is a problem of hand-eye coordination,and the global field of view of the patient’s lesion area cannot be obtained in real time.In recent years,HoloLens has been gradually applied to the emerging augmented reality surgical navigation system,but the existing HoloLens holographic space calibration methods mostly rely on the user’s subjective experience,and the overlay result does not consider the errors caused by user behavior and changes in the surrounding environment.In this regard,an accurate fusion overlay method is proposed based on HoloLens for augmented reality surgical navigation system.Use binocular cameras and visual markers to complete real-time tracking of target objects in real space.Based on the principle of hand-eye calibration,a stable HoloLens offline calibration method is proposed.By collecting the coordinate system conversion relationship between HoloLens holographic space and real space for many times,the mapping relationship between holographic space and real space can be accurately solved.After establishing the posture matrix transfer relationship of the fusion overlay,users can quickly perform online corrections to achieve accurate fusion overlay of the virtual image and the patient’s lesion area.The average time of HoloLens offline calibration is 4 min,and the average time of user online calibration is 6.0 s.Multiple volunteers wore HoloLens devices to perform fusion overlay accuracy verification experiments.The experimental results show that the average angle error and average distance error of HoloLens fusion overlay before correction are 1.01°and 3.95 mm,respectively,and the average distance error of fusion overlay after correction is 1.46 mm.The accurate fusion overlay method of the augmented reality surgical navigation system proposed in this paper can accurately calibrate the HoloLens holographic space.After online correction by the user,the virtual image can be fused with the

关 键 词：增强现实 HoloLens 离线标定 在线校正 手术导航系统 

分 类 号：TP391[自动化与计算机技术—计算机应用技术] R61[自动化与计算机技术—计算机科学与技术]