基于Spark Streaming的增量协同过滤算法  

Incremental Collaborative Filtering Algorithm Based on Spark Streaming

作　　者：曾志武 蔡明[1] ZENG Zhiwu;CAI Ming(College of Internet of Things,Jiangnan University,Wuxi 214122,China)

机构地区：[1]江南大学物联网工程学院,江苏无锡214122

出　　处：《软件导刊》2018年第6期88-91,共4页Software Guide

摘　　要：针对协同过滤算法处理大数据流时响应慢的缺陷,在改善推荐准确度的情况下,提出增量更新算法以加快响应速度,提高推荐系统性能。介绍了当前协同过滤算法以及KNN和Spark的相关知识,阐述了协同过滤算法的增量模型。采用Group Lens网站提供的Movie Lens数据集作为实验数据,应用Socket模拟流和Spark并行计算技术实现增量模型。实验结果显示,在保证推荐准确度的前提下,响应时间明显缩短,说明增量模型适合实时处理大数据流,可缓解数据处理不及时问题。Because of the slow response of collaborative filtering algorithm in dealing with large data streams,this paper presents an incremental updating algorithm to speed up the response times and improve the recommendation system performance under the condition of guaranteeing the accuracy of recommendation.Firstly,this paper presents the background and purpose of the study,and then introduces the current collaborative filtering algorithm and its related knowledge of KNN and Spark.Secondly,the incremental model of collaborative filtering algorithm is proposed.Finally,we used Movie Lens dataset provided by Group Lens website was used as the experimental data source,with Spark Stream to receive stream data from Socket and Spark to parallel computing increment data.The experimental results showed that in the case of reliable recommendation accuracy,response times is significantly improved and it proves that the incremental model proposed in this paper is very suitable for realtime processing of large data stream to alleviate the problem of no timely processing data.

关 键 词：协同过滤 推荐系统 增量计算 实时流计算 SPARK STREAMING 

分 类 号：TP312[自动化与计算机技术—计算机软件与理论]