大数据环境下的分布式数据流实时处理技术研究（英文）  

Study on real-time processing technology of distributed data flow in big data environment

作　　者：梁肖 孙丽[1] Xiao LIANG;Li SUN(Department of Computer Science,Hebei Professional College of Political Science and Low,Shijiazhuang 050061,China)

机构地区：[1]河北政法职业学院计算机系,石家庄050061

出　　处：《机床与液压》2019年第6期150-155,共6页Machine Tool & Hydraulics

基　　金：Research topic of Hebei Department of human resources and social security in 2016-Research on the transfer of employment channel based on large data in rural labor force(JRS-2014-1106);Research project of Hebei Professional College of Political Science and Low in 2016 "Research on the problems and Countermeasures of students’Entrepreneurship in E-commerce Major"(ZF20160613)~~

摘　　要：大数据背景下的海量分布式数据流处理技术要求较高的实时性和成功率。为了提高分布式环境下数据流实时处理的实时性,采用Hadoop分布式系统平台,提出了一种Map Reduce工作流实时调度优化方法。该方法采用2个相关参数来动态调整每个作业在队列中的重要程度,以便对工作流的优化做出动态的判断。此外,为了减少工作流整体的执行时间,对关键作业能否本地化进行判断和时间对比,从而尽量增加map任务的本地化。实验结果显示,提出的工作流实时调度优化方法在响应时间和调度成功率方面表现出较好的性能。Under the background of big data,massively distributed data stream processing technology requires high real-time and success rate.In order to improve the real-time performance of real-time processing of data streams in distributed environment,a real-time scheduling optimization method of Map Reduce workflow based on Hadoop distributed system platform was proposed.In addition,in order to reduce the overall execution time of the workflow,the proposed strategy judge whether the key jobs can be localized and compare with time,so as to maximize the localization of map tasks.The experimental results show that the proposed workflow real-time scheduling optimization method shows better performance in terms of response time and scheduling success rate.

关 键 词：大数据 MAP REDUCE 工作流 分布式 调度方法 

分 类 号：TN91[电子电信—通信与信息系统]