基于动态均衡技术的海量异构数据高并发可靠接入方法  

High concurrency and reliable access method for massive heterogeneous data based on dynamic balancing technology

作　　者：赵勋 周成胜 靳文京 刘晓曼 王桂温 Zhao Xun;Zhou Chengsheng;Jin Wenjing;Liu Xiaoman;Wang Guiwen(Institute of Security,China Academy of Information and Communications Technology,Beijing 100191,China)

机构地区：[1]中国信息通信研究院安全研究所,北京100191

出　　处：《网络安全与数据治理》2023年第12期60-66,共7页CYBER SECURITY AND DATA GOVERNANCE

基　　金：2022年工业和信息化部制造业专项项目(20230049)。

摘　　要：随着大数据时代的到来,海量异构数据的高并发可靠接入成为了一个亟待解决的问题。提出了一种基于动态均衡技术的海量异构数据高并发可靠接入方法。该方法采用去中心化的任务分配机制实现海量数据源接入;针对各类异构数据源设计了基于HTTPS、SFTP、Kafka等多样化采集手段及相应的节点分配和回收机制;采用动态负载均衡策略对采集资源进行实时调整,以适应不断变化的数据负载,实现高并发处理。该研究为实现海量异构数据的高效、可靠接入提供了一种有效的解决方法。With the arrival of the era of big data,the highly concurrent and reliable access of massive heterogeneous data has become an urgent problem.This paper proposes a high concurrent and reliable access method for massive heterogeneous data based on dynamic balance technology.The method adopts decentralized task allocation mechanism to access massive data sources.For various heterogeneous data sources,a variety of collection methods based on HTTPS,SFTP,Kafka,and corresponding node allocation and recovery mechanisms are designed.The dynamic load balancing strategy is used to adjust the collection resources in real time to adapt to the changing data load and achieve high concurrency processing.This research provides an effective solution for the efficient and reliable access of massive heterogeneous data.

关 键 词：海量异构数据 高并发 动态负载均衡策略 

分 类 号：TP393.08[自动化与计算机技术—计算机应用技术]