改进一致性哈希优化存储邮政数据算法的研究  

Research on improved consistency hash optimization algorithm for storing postal data

作　　者：李泽山 LI Zeshan(The Information Center of National Forestry and Grassland Administration,Beijing 100714,China)

机构地区：[1]国家林业和草原局信息中心,北京100714

出　　处：《现代电子技术》2024年第6期43-48,共6页Modern Electronics Technique

基　　金：内蒙古自然科学基金项目(2020MS06029);内蒙古自然科学基金项目(2021LHMS06013);内蒙古自然科学基金项目(2020LH06009);内蒙古自治区关键技术攻关计划项目(2020GG0165)。

摘　　要：随着电子商务不断发展,邮政快递行业数据日益增多,传统方式对于邮政数据存储的理论与方法都已无法满足需求。基于此情况,使用一致性哈希算法来解决存储系统的横向弹性扩展,结合一致性哈希的虚拟节点与加权轮询算法优化Hadoop平台下分布式文件系统(HDFS)存储策略,实现集群在同构与异构条件下的数据均衡效果。同时介绍集群节点数据转移思想,设计负载因子与系统自检周期,实现了集群动态权重的负载转移,并进行实验验证。实验结果表明,文章提出的改进算法与HDFS、普通一致性哈希相比,在不同条件下集群负载差值均有不同程度的提升,证明了该策略可以有效降低集群节点间负载差值。With the continuous development of e-commerce and the increasing number of data in the postal express industry,traditional methods are no longer meet the needs for the theories and methods of postal data storage.Based on this situation,the consistent hash algorithm is used to solve the horizontal elastic expansion of the storage system,and the storage strategy of distributed file system(HDFS) on Hadoop platform is optimized by combining consistent hashing with virtual nodes and weighted polling algorithm to realize the data balance in clusters under homogeneous and heterogeneous conditions.The concept of cluster node data transfer is introduced,and the load factors and system self check cycles is designed,so as to realize the load transfer of dynamic cluster weights,and conduct the experimental verification.The experimental results show that the proposed improved algorithm has different degrees of improvement in cluster load difference compared with HDFS and regular consistency hashing under different conditions,proving that this strategy can effectively reduce the load difference between cluster nodes.

关 键 词：数据存储 一致性哈希算法 加权轮询算法 分布式文件系统 负载均衡 异构集群 分配策略 

分 类 号：TN919-34[电子电信—通信与信息系统] TP319.9[电子电信—信息与通信工程]