基于Spark的混合推荐算法研究  

Research on hybrid recommendation algorithm based on Spark technology

作　　者：胡德敏[1,2] 龚燕[1] 

机构地区：[1]上海理工大学光电信息与计算机工程学院,上海200093 [2]上海理工大学计算机软件技术研究所,上海200093

出　　处：《计算机应用研究》2017年第12期3585-3588,共4页Application Research of Computers

基　　金：国家自然科学基金资助项目(61170277;61472256);上海市教委科研创新重点资助项目(12zz137);上海市一流学科建设资助项目(S1201YLXK)

摘　　要：随着电子商务的发展,传统的单机计算模型难以满足海量数据的实时推荐需求,基于协同过滤的推荐算法的缺陷也越来越明显。为此,提出一种利用Spark计算模型实现分布式推荐的方法。该推荐方法采用基于谱聚类和朴素贝叶斯的混合推荐算法,同时使用增量式更新,在不全部重新训练模型的基础上,对模型进行局部修改。实验结果表明,较传统的单机模式推荐算法,基于Spark计算模型的分布式推荐算法在一定程度上克服了数据稀疏性,提高了系统的可扩展性,降低了系统的响应时间。Due to the development of e-commerce,traditional stand-alone model is difficult to meet the needs of massive data in real-time recommendation. And collaborative filtering-based recommender system has become increasingly evident. Therefore,this paper proposed a distributed recommendation method based on Spark computing model,which the theory was based on spectral clustering and Naive Bayes. In addition,the hybrid method used the increment update schemes to refresh the ratings and improved the precision of the system,without all the re-training model. The experimental results demonstrate that to be compared with traditional stand-alone mode recommendation algorithm,the distributed recommendation algorithm overcomes sparsity and scalability problem to a certain extent and has higher scalability and reduces the response time of the system.

关 键 词：推荐算法 分布式计算 SPARK 增量式更新 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]