视听同步的细粒度脚步音效合成方法  

Fine⁃grained footsteps sound synthesis for audiovisual synchronisation

作　　者：刘子航 齐秋棠 程皓楠 崔健[3] 叶龙[1,2] Liu Zihang;Qi Qiutang;Cheng Haonan;Cui Jian;Ye Long(School of Data Science and Intelligent Media,Communication University of China;State Key Laboratory of Media Convergence and Communication,Communication University of China;Xinhua News Agency)

机构地区：[1]中国传媒大学数据科学与智能媒体学院,北京100024 [2]中国传媒大学媒体融合与传播国家重点实验室,北京100024 [3]新华通讯社,北京100803

出　　处：《现代电影技术》2023年第12期4-10,56,共8页Advanced Motion Picture Technology

基　　金：国家自然科学基金青年项目《基于数据与机理融合的交互感环境声合成理论与方法研究》(62201524)。

摘　　要：电影后期音效制作合成目前仍主要依赖手动操作,其人力与资源成本较高。现有智能拟音技术由于存在合成声音缺乏细粒度内容和真实感不足等问题,难以满足实际电影后期音效制作要求。针对上述问题,本文提出了一种细粒度脚步音效的自动合成方法,充分利用视觉画面信息,以实现视听同步且内容匹配度高的脚步声音效合成。具体而言,本文采用数据驱动的视听跨模态生成方法,深入学习视听时序关联,以实现视听同步。随后,为了进一步丰富合成脚步声音的内容细粒度,对视觉画面中的地面材质和人物运动信息进行深入分析,并构建与声音之间的规则连接。实验证明,本文的方法能够合成与视觉信息匹配的时间同步且内容合理的脚步音效,实现了脚步音效的自动化生成,提升了视听的真实感。Film post⁃production sound effects are currently mainly produced by manual operation,which is costly and time⁃consuming.Existing intelligent Foley technologies cannot meet the realistic demands of film post⁃production sound effects due to the lack of fine⁃grained content and realism in synthesized sound.To address these challenges,this paper pro⁃poses a fine⁃grained footsteps sound synthesis method for audiovisual synchronization,which leverages the visual image information to achieve synchronized and content⁃matched footsteps sound effects.Specifically,this paper adopts a data⁃driven approach to audiovisual cross⁃modal generation to learn the audio⁃visual temporal correlations and achieve audio⁃vi⁃sual synchronization.Furthermore,to enhance the content granularity of the synthesized footsteps sounds,the study deeply analyzes the ground material and character motion information in the visual images,and mapped them to the corre⁃sponding sounds with particular rules.Experiments show that the proposed method can synthesize time⁃synchronized and content⁃reasonable footsteps sound effects that match the visual information,as well as realize the automated generation of footstep sound effects,to improve the audiovisual realism.

关 键 词：电影音效制作 智能化拟音 脚步音效合成 跨模态视听生成 

分 类 号：TP37[自动化与计算机技术—计算机系统结构]