Kafka中改进型Partition过载优化算法  

Improved Partition Overload Optimization Algorithm in Kafka

作　　者：颜晓莲 章刚 邱晓红 YAN Xiao-lian;ZHANG Gang;QIU Xiao-hong(School of Software Engineering(Nanchang),Jiangxi University of Science and Technology,Nanchang 330013,China;Jiangxi Peking University Science Park,Nanchang 330013,China)

机构地区：[1]江西理工大学软件工程学院(南昌),江西南昌330013 [2]江西北大科技园,江西南昌330013

出　　处：《计算机技术与发展》2020年第12期88-91,共4页Computer Technology and Development

基　　金：江西省教育科技项目(GJJ170571)。

摘　　要：Kafka作为一种发布-订阅机制的高吞吐量分布式消息系统,广受业界关注。随着应用场景垂直化、多样化,Kafka现有的技术体系面临挑战。Partition过载问题(POP)指消息分发、消息订阅等操作引起Partition过度服务,并影响到物理载体Broker的性能。该问题是由Kafka中Partition文件配置管理的被动、僵化及孤立等不足所导致。针对此,提出一种改进型Partition过载优化算法(IPOOA)。该算法通过即时服务耗量、Partition相似度和配置文件自动修改相结合,实现消息分发预测以及消息分发与文件配置管理协同,从而可有效缓解Partition过载问题出现。实验从Kafka集群CPU使用率、Kafka服务延时率、Kafka系统收敛延时比等几个方面验证了算法的有效性及合理性。Kafka,as a high-throughput distributed message system with publish subscribe mechanism,is widely concerned by the industry.Existing technology system of Kafka is facing challenges by the vertical and diversified application scenarios.Partition overload problem(POP)refers to the problem that operations such as message distribution and message subscription cause excessive service of partition which affects the performance of physical carrier broker.This problem is caused by the passivity,rigidity and isolation of partition file configuration management in Kafka.Therefore,we propose an improved optimization algorithm of partition overload(IPOOA).This algorithm combines the instant service consumption,partition similarity and automatic modification of configuration file to realize the prediction of message distribution and the coordination of message distribution and file configuration management,which can effectively alleviate the problem of partition overload.The experiment verifies the validity and rationality of the algorithm from several aspects,such as the CPU utilization rate of Kafka cluster,Kafka service delay rate,Kafka system convergence delay rate,etc.

关 键 词：分布式消息系统 Kafka Partition过载问题 协同管理 Broker性能 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]