异构Hadoop集群下的比例数据分配策略  

PROPORTIONAL DATA PLACEMENT STRATEGY IN HETEROGENEOUS HADOOP CLUSTERS

作　　者：魏文娟[1] 王黎明[1] 

机构地区：[1]郑州大学信息工程学院,河南郑州450052

出　　处：《计算机应用与软件》2015年第6期316-319,共4页Computer Applications and Software

摘　　要：针对异构Hadoop环境下仍采用均等的数据分配方法将严重降低MapReduce的性能,提出比例数据分配策略。通过计算异构集群中各节点的计算比率,将已经分割好的数据块重新进行组合,形成数个按比例划分的数据块。每个节点根据自身性能来选择所分配和存储的数据块,从而使异构Hadoop集群中各节点处理数据的时间大致相同,降低节点之间数据的移动量。实验验证了提出的比例数据分配方法可以有效地提高MapReduce的性能,并使数据负载均衡。We present a proportional data placement strategy for the issue that to still use equal data placement in heterogeneous Hadoop environment will seriously debase the performance of MapReduce.By computing the computation rate of each node in heterogeneous clusters, we recombine the segmented data block to form a number of data blocks proportionally partitioned.According to its own characteristic each node selects the distributed and stored data block,so that every node in heterogeneous Hadoop clusters processes data in basically same time, and thus the amount of data transfer between nodes is reduced.It is verified by experiment that the proposed proportional data placement stra-tegy can improve MapReduce performance effectively and makes the data load in equilibrium.

关 键 词：异构Hadoop 集群 MAPREDUCE 数据转移 比例数据分配 计算速率 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]