基于余弦向量法的Web数据并行抓掘系统  

Parallel Crawling System for Web Data Based on Cosine Vector

作　　者：徐文杰[1] 陈庆奎[1] 

机构地区：[1]上海理工大学计算机与电气工程学院,上海200093

出　　处：《计算机工程》2009年第7期64-67,共4页Computer Engineering

基　　金：国家自然科学基金资助项目(60573108);上海教委发展基金资助项目(06QZ002;07ZZ92);上海教委科研创新基金资助重点项目(08ZZ76);上海市重点学科建设基金资助项目(S30501)

摘　　要：为了提高Web海量数据的抓掘效率,引入并行机群抓掘机制。为使机群中每个计算节点的能力得到充分发挥,应用向量度量技术解决抓取任务和计算节点能力匹配的问题。对抓取任务向量、计算节点向量进行定义,提出余弦向量匹配算法,描述相关并行算法。理论分析和实验表明,基于余弦向量匹配算法的挖掘任务分配模型具有良好的分配适应性和负载平衡性。This paper proposes a parallel cluster crawling model to improve the mining efficiency of massive data on Web. For fully using of the ability of parallel nodes in computer duster, a vector measurement technology is introduced to solve the matching problem between crawling task and computer node. After giving the definitions of crawling task vector and computer node vector, cosine vector similarity formula is described, and the parallel crawling algorithms is designed. Experimental results show that the system is effective in distribution adaptability and load balance.

关 键 词：并行抓取 余弦向量法 计算机机群 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]