基于迁移学习与强化学习的自动音频标注系统  

Automated Audio Caption System Based on Transfer Learning and Reinforcement Learning

作　　者：陈耕耘 李圣辰 邵曦[1] 梅昕浩 刘徐博 黄秋实 王文武 CHEN Gengyun;LI Shengchen;SHAO Xi;MEI Xinhao;LIU Xubo;HUANG Qiushi;WANG Wenwu(College of Telecommunications and Information Engineering,Nanjing University of Posts and Telecommunications,Nanjing,Jiangsu 210003,China;School of Advanced Engineering,Xi'an Jiaotong-Liverpool University,Suzhou,Jiangsu 215123,China;Electrical and Electronic Engineering,University of Surrey,Guildford,Surrey,England GU24BB)

机构地区：[1]南京邮电大学通信与信息工程学院,江苏南京210003 [2]西交利物浦大学先进工程学院,江苏苏州215123 [3]萨里大学电子与电气工程学院,英国萨里郡吉尔福德GU24BB

出　　处：《复旦学报（自然科学版）》2022年第5期520-526,共7页Journal of Fudan University：Natural Science

基　　金：国家自然科学基金(61936005,61872199,61872424)。

摘　　要：自动音频标注是让计算机为一段音频自动生成标注语句来对该片段进行描述的任务。针对当前音频标注模型不够有效,且模型训练目标与评价指标得分之间不一致的问题,本文提出了一种基于CNNTransformer的编码器解码器结构,并采用强化学习进行微调的方法,同时采用了迁移学习为编码器提供一个预训练网络来初始化参数。在Clotho数据集上的实验结果表明,与基线系统相比,该方法能够提升模型的有效性,而通过强化学习对模型进行微调同样大幅提高了优化对象指标的得分。Automated audio captioning task aims at generating text description of a piece of audio.This system based on an encoder-decoder architecture,consisting of a Convolutional Neural Network(CNN)encoder and a Transformer decoder.In order to solve the problem of the mismatch between evaluation metrics and the loss function,reinforcement learning is investigated for generating more accurate caption.Furthermore,the system has a pre-trained encoder via transfer learning.Clotho dataset was chosen as our dataset.The results show that both techniques can further improve the performance of the captioning system.

关 键 词：自动音频标注 强化学习 迁移学习 深度学习 

分 类 号：TP391.9[自动化与计算机技术—计算机应用技术]