基于节点负载的数据动态分区  

Dynamic Data Partition Based on Node Load

作　　者：孟令伍 杨阳朝 黄晓明 练丽萍 MENG Ling-Wu;YANG Yang-Chao;HUANG Xiao-Ming;LIAN Li-Ping(Nanjing Rice Net Information Technology Research Institute Co.Ltd.,Nanjing 210014,China;Shenzhen CyberArey Network Technology Co.Ltd.,Shenzhen 518038,China;CETC New Smart City Research Institute Co.Ltd.,Shenzhen 518026,China)

机构地区：[1]南京莱斯网信技术研究院有限公司,南京210014 [2]深圳市网联安瑞网络科技有限公司,深圳518038 [3]中电科新型智慧城市研究院有限公司,深圳518026

出　　处：《计算机系统应用》2021年第12期299-307,共9页Computer Systems & Applications

摘　　要：一种基于节点负载的数据动态分区系统,主要考虑节点CPU、内存、带宽负载情况,首先采用二次平滑法预测节点的负载,再结合AHP和熵值指标权重法得到每个节点的处理能力,最后针对不同应用场景动态地调整系统的负载均衡性,提高应用的响应速度;该系统主要包括负载监测采集、预测、数据预分区、数据迁移等模块.由于分布式环境存在节点资源的异构性,为了数据分析计算过程中减少节点之间数据的传输,充分利用节点计算资源,通过负载均衡性提高应用分析的并行计算速度.为此,本文提出一种基于节点负载的数据动态分区机制和策略来改善系统负载均衡性及提高应用的响应速度,辅助相关工作人员完成决策.本论文结合Spark和Elasticsearch集成的数据分析应用场景进行测试.A dynamic data partition system based on node load mainly considers the load of CPU,memory,and bandwidth of nodes.It first uses the quadratic smoothing method to predict the load of nodes,then combines AHP and entropy index weight to get the processing capacity of each node,and finally dynamically adjusts the load balance of the system for different application scenarios to improve the response speed of applications.It includes the modules of load monitoring and collection,prediction,data pre-partition,and data migration.Given the heterogeneity of node resources in a distributed environment,it aims to reduce the data transmission between nodes in the process of data analysis and calculation,make full use of node computing resources,and improve the parallel computing speed of application analysis through load balancing.Therefore,this study proposes a dynamic partition data mechanism based on node load to improve the system load balance and application response speed and assist the relevant staff in making the decision.This study combines data analysis application scenarios integrating Spark and Elasticsearch for testing.

关 键 词：负载均衡 动态分区机制 SPARK Elasticsearch 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论] TP333[自动化与计算机技术—计算机科学与技术]