Generative Multi-Modal Mutual Enhancement Video Semantic Communications  

作　　者：Yuanle Chen Haobo Wang Chunyu Liu Linyi Wang Jiaxin Liu Wei Wu 

机构地区：[1]The College of Communication and Information Engineering,Nanjing University of Posts and Telecommunications,Nanjing,210023,China [2]The College of Science,Nanjing University of Posts and Telecommunications,Nanjing,210023,China

出　　处：《Computer Modeling in Engineering & Sciences》2024年第6期2985-3009,共25页工程与科学中的计算机建模（英文）

基　　金：supported by the National Key Research and Development Project under Grant 2020YFB1807602;Key Program of Marine Economy Development Special Foundation of Department of Natural Resources of Guangdong Province(GDNRC[2023]24);the National Natural Science Foundation of China under Grant 62271267.

摘　　要：Recently,there have been significant advancements in the study of semantic communication in single-modal scenarios.However,the ability to process information in multi-modal environments remains limited.Inspired by the research and applications of natural language processing across different modalities,our goal is to accurately extract frame-level semantic information from videos and ultimately transmit high-quality videos.Specifically,we propose a deep learning-basedMulti-ModalMutual Enhancement Video Semantic Communication system,called M3E-VSC.Built upon a VectorQuantized Generative AdversarialNetwork(VQGAN),our systemaims to leverage mutual enhancement among different modalities by using text as the main carrier of transmission.With it,the semantic information can be extracted fromkey-frame images and audio of the video and performdifferential value to ensure that the extracted text conveys accurate semantic information with fewer bits,thus improving the capacity of the system.Furthermore,a multi-frame semantic detection module is designed to facilitate semantic transitions during video generation.Simulation results demonstrate that our proposed model maintains high robustness in complex noise environments,particularly in low signal-to-noise ratio conditions,significantly improving the accuracy and speed of semantic transmission in video communication by approximately 50 percent.

关 键 词：Generative adversarial networks multi-modal mutual enhancement video semantic transmission deep learning 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]