一种跨HDFS集群的文件资源分布式高效存储方法  

A distributed and efficient storage method for file resources across HDFS clusters

作　　者：戴威 DAI Wei(Jiangsu Electric Power Company Research Institute,Nanjing 211103,China)

机构地区：[1]国网江苏省电力有限公司电力科学研究院

出　　处：《电子设计工程》2019年第21期14-17,22,共5页Electronic Design Engineering

基　　金：江苏省科技厅项目(047800HK32180006)

摘　　要：针对传统文件资源存储方法存在存储效率低的问题,提出了一种跨HDFS集群的文件资源分布式高效存储方法。依据HDFS集群文件资源特征及存储原理,分析资源分布式存储的逻辑关系,保证文件资源存储环境安全。依据标准Hadoop布局方案,保持设备物理特性,对资源分布进行优化配置,实现文件资源存储自定义功能。通过文件合并、映射计算元数据在名字节点中内存损耗情况,设计高效存储过程,由此完成集群文件资源的分布式高效存储。在集群环境拓扑结构中进行实验验证,通过实验结果可知,该方法存储效率最高为98%,能够为文件资源安全存储提供技术支持。Aiming at the low storage efficiency of traditional file resource storage methods,a distributed and efficient storage method for file resources across HDFS clusters is proposed.According to the characteristics and storage principle of HDFS cluster file resources,the logical relationship of distributed storage of resources is analyzed to ensure the security of file resources storage environment.According to the standard Hadoop layout scheme,the physical characteristics of the device are maintained,the resource distribution is optimized,and the file resource storage customization function is realized.The memory loss of metadata in name nodes is calculated by file merging and mapping,and efficient stored procedures are designed to achieve distributed and efficient storage of cluster file resources.The experimental results in the cluster environment topology show that the storage efficiency of this method is up to 98%,which can provide technical support for the safe storage of file resources.

关 键 词：跨HDFS集群 文件资源 分布式 高效存储 优化配置 

分 类 号：TN18[电子电信—物理电子学]