基于迭代MapReduce的混合云大数据分析  

Big data analysis based on iterative MapReduce for hybrid cloud

作　　者：颜烨[1] 张学文[2] 王立婧 YAN Ye;ZHANG Xue-wen;WANG Li-jing(College of Electrical Information,City College of Science and Technology,Chongqing University,Chongqing 402167,China;School of Mechanical Engineering,Beihua University,Jilin 132021,China;College of Humanities,City College of Science and Technology,Chongqing University,Chongqing 402167,China)

机构地区：[1]重庆大学城市科技学院电气信息学院,重庆402167 [2]北华大学机械工程学院,吉林吉林132021 [3]重庆大学城市科技学院人文学院,重庆402167

出　　处：《计算机工程与设计》2021年第4期1028-1035,共8页Computer Engineering and Design

基　　金：吉林省自然科学基金项目(20150101025JC);高档数控机床科学与基础制造装备科技重大专项基金项目(2015ZX040003002);2018年重庆市本科高校大数据智能化类特色专业建设基金项目(渝教高发[2018]12号)。

摘　　要：现有混合云平台运行大数据分析等数据密集型工作负载存在数据迁移开销大、执行时间长等问题,提出基于迭代MapReduce的大数据分析方法。提出一种数据存储和数据迁移机制,迭代时将初始不变量从私有云迁移至公有云,无需修改MapReduce框架或底层存储层;利用随机森林估计所提迭代MapReduce应用程序所需的计算时间。OpenStack混合云实验结果表明,相比传统方案,提出方法仅增加了初始迭代的运行时间,最终完成时间比传统方法节约12.6%以上。此外,提出的性能预测方法的误差率可保持在19.54%以内。Running data-intensive workloads such as large data analysis in a current hybrid cloud framework has the shortcomings such as high data migration and long execution time.A big-data analysis implementation method based on iterative MapReduce was proposed.A data storage and migration mechanism in which the initial invariants were migrated from the private cloud to the public cloud in the first iteration was proposed,which had the advantage of no modification of MapReduce framework or underl-ying storage layer.In addition,random forest(RF)algorithm was used to estimate the computational time required for the proposed iterative MapReduce application.A hybrid cloud example based on OpenStack shows that,compared with the traditional methods,the proposed method just increases the running time of initial iteration while shortens the final iterative time by more than 12.6%.In addition,the error rate of the proposed performance prediction method maintains within 19.54%.

关 键 词：混合云 大数据分析 迭代MapReduce 数据迁移 随机森林算法 性能预测 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]