基于异常检测模型的异构环境下MapReduce性能优化  

MapReduce performance optimization based on anomaly detection model in heterogeneous cloud environment

作　　者：侯佳林[1] 王佳君[1] 聂洪玉[1] 

机构地区：[1]西南交通大学信息科学与技术学院,成都610031

出　　处：《计算机应用》2015年第9期2476-2481,共6页journal of Computer Applications

基　　金：中央高校基本科研业务费专项资金专题研究项目(SWJTU11ZT08);国家语委"十二五"科研规划项目(YB125-49)

摘　　要：针对"落伍者"的选择问题,提出利用故障诊断领域内通常使用的异常检测模型来选择"落伍者"的方法。首先,利用异常检测算法来发现集群中的"慢节点";然后改进MapReduce任务分配算法和推测执行算法,不再给"慢节点"分配任务并将"慢节点"中的任务分配至有空闲任务槽的正常节点中。在改进的推测执行算法中,因相同网段内的节点通常物理邻近,可提高数据传输速度,首次将"慢节点"中的任务分配至同网段的正常节点中,以便数据传输。实例验证结果表明,使用异常检测算法后可迅速检测出异常节点,且与Hadoop-LATE算法相比,处理相同任务量可缩短集群17%的任务处理时间,说明所提算法在集群整体性能优化中表现优异。To effectively select the straggler machines, an anomaly detection model generally adopted in failure analysis was proposed. Firstly, an anomaly detection algorithm was employed to detect the slow nodes in the cluster. Secondly, task assignment algorithm and speculative execution algorithm were improved to stop assigning new tasks to slow nodes and these tasks were assigned to normal nodes with idle slots. In the improved speculative execution, it was for the first time that those tasks in slow nodes were transferred into the normal nodes in the same network segment, since data transferring can be physically accelerated in one network segment. The experimental results demonstrate that the straggler machines are quickly detected after running the anomaly detection algorithm. Compared with the algorithms in Hadoop-LATE, 17% of the processing time can be saved when the same amount of the tasks are processed, which concludes that the proposed algorithm is more suitable for improving the overall performance of the clusters.

关 键 词：异常检测 MapReduce性能优化 推测执行 异构环境 

分 类 号：TP302[自动化与计算机技术—计算机系统结构]