一种基于Spark时效化协同过滤推荐算法  

A Real-time Collaborative Filtering Recommendation Algorithm Based on Spark

作　　者：徐新瑞[1] 孟彩霞[1] 周雯[1] 刘盈[1] 

机构地区：[1]西安邮电大学计算机学院,陕西西安710121

出　　处：《计算机技术与发展》2015年第6期48-55,共8页Computer Technology and Development

基　　金：国家自然科学基金资助项目(61105064);陕西省自然科学基金资助项目(2014JM8303);陕西省教育专项科研计划资助项目(11JK0988);西安邮电大学研究生创新基金项目(ZL2013-42)

摘　　要：针对传统的批量学习的基于模型的协同过滤算法对新用户(物品)更新缓慢,模型重训练成本高且扩展性不足,对噪音数据的处理有待提高,尤其是随着数据量的增长和时效性要求越来越高,挖掘其中的知识变得越来越困难等问题,对置信权重在线协同过滤算法进行改进。引入自适应软边缘,提出二阶在线优化方法处理在线协同过滤中问题的新算法(Soft Confidence Weighted Online Collaborative Filtering,SCWOCF),并在Spark流处理推荐框架下利用四组真实数据与相关算法作对比测试。实验结果表明,新算法能够及时处理用户(物品)的动态变化,并提升推荐的实时性和准确性,降低计算成本,对噪声数据健壮性更强。Focused on some drawbacks of traditional collaborative filtering algorithms based on model of batch learning,such as updating slowly for new users or items,highly retraining cost and expanding difficultly,and handling noise data need to be improved,especially, being more and more difficult for knowledge mining with growing data and the requirement of real-time,the online collaborative filtering algorithm of confidence weighted is improved. In order to solve these problems, a new algorithm named SCWOCF （ Soft Confidence Weighted Online Collaborative Filtering） was proposed. In this algorithm,the adaptive soft margin was added and the second order online optimization methodology was used to solve online collaborative filtering problems. Finally, several experiments with four real-world datasets was conducted compared with some similar algorithms on the Spark stream processing recommendation framework. The results show that the new algorithm can timely handle dynamic change of users and items,promoting the real-time and accuracy of recommenda-tion,reducing cost of computation,increasing robustness to noise data.

关 键 词：在线学习 自适应软边缘 软置信权重 二阶协同过滤 推荐系统 HADOOP SPARK on YARN 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]