Yarn架构下基于GA的Web日志挖掘技术  

Web log mining technology using GA based on Yarn

作　　者：李静梅[1] 张宝权[1] 丁楠[2] 

机构地区：[1]哈尔滨工程大学计算机科学与技术学院,哈尔滨150001 [2]清华大学计算机系,北京100084

出　　处：《计算机应用研究》2014年第11期3388-3391,共4页Application Research of Computers

基　　金：国家自然科学基金资助项目(61003036);黑龙江省自然科学基金资助项目(F201124);黑龙江省教育厅科学技术研究基金资助项目(12513048)

摘　　要：提出了一种面向TB级别日志文件挖掘需求的日志挖掘技术。采用MapReduce编程模型设计并实现了一种基于子种群联姻策略的GA,并部署到Yarn架构上,使Yarn架构与GA有效结合。在算法Partition阶段,采用Round-Robin策略代替原有的哈希方法,使各个子种群基因更加趋向均匀分布,增加了子种群收敛效率和结果准确性,同时均衡各个节点运行负载,提高了系统性能。经测试,应用该技术的挖掘结果平均准确度达到93%以上,效率提升接近33%。This paper proposed a Web log mining technology orienting the demand of mining for TB-level log files.It designed and implemented a genus of GA which deployed on the Yarn using the MapReduce programing model to combine the GA with the Yarn effectively.In addition,took the Round-Robin strategy instead of the hash method during the Partition stage of Ma-pReduce in order to distribute the gene among the sub populations evenly,increased the populations’rate of convergence and validity of result,and balanced the workload amid all of the nodes at the same time.Finally,the test shows that the rate of the validate result using the technology,which improves the efficiency by 33%,the average accuracy is more than 93%.

关 键 词：Yarn架构 日志挖掘 遗传算法 并行计算 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]