基于云平台的电信数据仓库文件备份和分布动态调整算法  

File Replication and Location Dynamic Adjustment Algorithm for Telecom Data Warehouse Based on Cloud Platform

作　　者：杨苗苗[1] 李跃辉[1] 刘静[1] 许静[1] 

机构地区：[1]南京邮电大学通信与信息工程学院,江苏南京210003

出　　处：《南京邮电大学学报（自然科学版）》2014年第1期111-115,共5页Journal of Nanjing University of Posts and Telecommunications：Natural Science Edition

摘　　要：Hadoop利用统一的备份机制来确保数据的可用性和容错性。然而对于重要的文件块,只能通过修改配置文件来设置副本数量。通过分析电信业数据仓库文件的访问情况,提出文件备份与存放动态调整算法(BPDA)来提高数据局部性,以减少数据传输带来的I/O和网络消耗,提高查询效率。运用多副本一致性哈希算法,概率采样和竞争力弱化算法,得到每一个文件的最佳备份数和备份位置。实验结果验证了该算法可以提高查询关联效率。Hadoop uses a uniform data replication scheme to ensure data availability and fault tolerance.For an important data block,the number of replicates can only be set by modifying the configuration file.Through the analysis of file access of telecom data warehouse,a file replicate and location dynamic adjustment algorithm,called the BPDA is proposed to improve the data locality,thus it reduces the I/O and network overhead due to data transmission,and enhances the efficiency of query processing.Using multi-copy consistency Hash algorithm,probability sampling and competitive aging algorithm,the optimized number and the storage location are achieved for each file.Experimental results show that BPDA can improve the efficiency of join query processing.

关 键 词：HADOOP MAPREDUCE 备份 关联查询处理 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]