时空联合视差优化的立体视频重定向  

Optimizing spatiotemporal disparities for stereoscopic video retargeting

作　　者：金康俊 柴雄力 邵枫[1] Jin Kangjun;Chai Xiongli;Shao Feng(Faculty of Information Science and Engineering,Ningbo University,Ningbo315211,China)

机构地区：[1]宁波大学信息科学与工程学院,宁波315211

出　　处：《中国图象图形学报》2022年第2期614-627,共14页Journal of Image and Graphics

基　　金：国家自然科学基金项目(62071261)。

摘　　要：目的智能适配显示的图像/视频重定向技术近年受到广泛关注。与图像重定向以及2D视频重定向相比,3D视频重定向需要同时考虑视差保持和时域保持。现有的3D视频重定向方法虽然考虑了视差保持却忽略了对视差舒适度的调整,针对因视差过大和视差突变造成视觉不舒适度这一问题,提出了一种基于时空联合视差优化的立体视频重定向方法,将视频视差范围控制在舒适区间。方法在原始视频上建立均匀网格,并提取显著信息和视差,进而得到每个网格的平均显著值;根据相似性变化原理构建形状保持能量项,利用目标轨迹以及原始视频的视差变化构建时域保持能量项,并结合人眼辐辏调节原理构建视差舒适度调整能量项;结合各个网格的显著性,联合求解所有能量项得到优化后的网格顶点坐标,将其用于确定网格形变,从而生成指定宽高比的视频。结果实验结果表明,与基于细缝裁剪的立体视频重定向方法对比,本文方法在形状保持、时域保持及视差舒适度方面均具有更好的性能。另外,使用现有的客观质量评价方法对重定向结果进行评价,本文方法客观质量评价指标性能优于均匀缩放和细缝裁剪的视频重定向方法,时间复杂度较低,每帧的时间复杂度至少比细缝裁剪方法降低了98%。结论提出的时空联合的视差优化方法同时在时域和舒适度上对视差进行优化,并考虑了时域保持,具有良好的视差优化与时域保持效果,展现了较高的稳定性和鲁棒性。本文方法能够用于3D视频的重定向,在保持立体视觉舒适性的同时适配不同尺寸的3D显示屏幕。ObjectiveIn recent years,with the rapid development of the digital video photography,people pay more attention to the imaging quality of videos with the increasing demand for the immersive experience.Therefore,it is a meaningful challenge to adjust stereoscopic content to the required size to accommodate the different resolutions of 3 D display devices.Stereoscopic video retargeting is quite different from image retargeting and 2 D video retargeting aims to minimize the shape distortions and optimize disparities with temporal coherence in resizing a stereoscopic video.More constraints of 3 D video retargeting should be considered,such as,temporal coherence and depth information.However,the existing 3 D video retargeting methods usually consider minimizing the depth distortion but not take the visual experience.Many studies show that the unsuitable disparity causes the visual discomfort and visual fatigue.As a result,someone does not have a great experience in 3 D video watching.To solve this problem,our study proposes a method to control the disparity into a comfortable range after 3 D video retargeting.Our method considers disparity remapping with temporal coherence and angular parallax.Furthermore,retargeting video may shaky and flickering because of the incoherence between frames.Mesh motion trajectory is utilized to constrain the retargeting video for minimizing the shaking.Considering the above factors,a stereoscopic video retargeting method via mesh warping in optimizing spatiotemporal disparities is proposed in this paper.MethodFirst,the mesh in first frame is constructed,and the stereoscopic saliency based on disparity and edge information is estimated.Then,the mean saliency of each grid for mesh importance is computed.Next,the vertex of the mesh in the original video is traced to obtain the vertex trajectories according to the optic flow method.Temporal coherence between the original and retargeted videos can be established by the motion trajectories.Shape distortion is of great importance in affecting t

关 键 词：立体视频重定向 网格形变 时空视差优化 视频时间一致性 立体视觉舒适度 立体显著 

分 类 号：TN919.81[电子电信—通信与信息系统]