基于迁移学习的个性化循环神经网络语言模型  

Personalized recurrent neural network language model based on transfer learning

作　　者：徐萍[1] 吴超 胡峰俊[1] 吴凡[1] 林建伟 刘静静[1] Xu Ping;Wu Chao;Hu Fengjun;Wu Fan;Lin Jianwei;Liu Jingjing(College of Information and Science Technology,Zhejiang Shuren University,Hangzhou 310015,China;College of Computer Science and Technology,Zhejiang University,Hangzhou 310058,China)

机构地区：[1]浙江树人大学信息科技学院,浙江杭州310015 [2]浙江大学计算机科学与技术学院,浙江杭州310058

出　　处：《南京理工大学学报》2018年第4期401-408,共8页Journal of Nanjing University of Science and Technology

基　　金：浙江省自然科学基金(LY14F020008);浙江省教育厅研究计划(Y201329701);国家自然科学基金(51675490);浙江省公益技术应用研究计划(2016C31116;2017C31050);浙江树人大学中青年学术团队项目

摘　　要：针对在小数据集上开发个性化语言模型仍存在的障碍,提出基于迁移学习的个性化循环神经网络语言模型。设计了基于预训练词向量、预训练电影剧本数据集、基于参数微调和特征提取分类器的迁移学习训练模式,在小数据集上建立了具有较高辨识度的个性化语言模型,降低了模型的困惑度,改进了模型的性能。模型的实验以电视剧Seinfeld角色为基础。结果表明:该模型在特定角色测试数据集上的困惑度比其他角色数据集平均低17.65%,证明其已经学会了该角色的个性化风格;迁移学习使得模型最低困惑度平均降低了36.38%,较好地解决了基于小数据集开发个性化语言模型存在的障碍问题。There are obstacles in the development of personalized language models on small data sets.A personalized recurrent neural network language model based on transfer learning is proposed.By designing a novel transfer learning training patterns based on pre-trained word vector,pre-trained external data,parameter fine-tuning and feature extraction classifier,a personalized language model with high degree of recognition is established on small data sets,both reducing the perplexity and improving the performance of the model.The experiment is conducted on the TV series Seinfeld’s roles.The results show that the perplexity of the role on the specific character test data set is 17.65%lower than that on other character data sets,which proves that the developed model has learned the personalized style of the character.It is demonstrated that the minimum perplexity of the model is reduced by 36.38%on an average through transfer learning,which proves that the developed model solves the obstacles mentioned above.

关 键 词：语言模型 个性化 循环神经网络 迁移学习 小数据集 预训练词向量 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]