一种基于LRFU缓存替换策略的HDFS客户端本地缓存设计与实现  

DESIGN AND IMPLEMENTATION OF AN HDFS CLIENT LOCAL CACHE BASED ON LRFU CACHE REPLACEMENT POLICY

作　　者：谢磊 吴建华[1] 廖卓凡[1] 罗可[1] Xie Lei;Wu Jianhua;Liao Zhuofan;Luo Ke(College of Computer and Information Engineering, Changsha University of Science and Technology, Changsha 410004 ,Hunan, China)

机构地区：[1]长沙理工大学计算机与通信工程学院,湖南长沙410004

出　　处：《计算机应用与软件》2018年第5期15-20,94,共7页Computer Applications and Software

基　　金：国家自然科学基金项目(61402056)

摘　　要：Hadoop底层基础存储框架Hadoop分布式文件系统HDFS(Hadoop Distributed File System)是采用基于Master/Slave主从构架设计的分布式文件系统。Name Node负责管理所有文件的元数据和处理客户端对文件的访问。随着应用数据的不断增加,Data Node的不断加入,Name Node维护文件元数据、处理Hadoop应用的读取和删除等文件操作的工作量就会随之增加。Name Node的性能成为整个HDFS的瓶颈。提出一种思想,即将Name Node部分任务交与HDFS客户端完成,使每个HDFS客户端在某些功能上成为Name Node。用LRFU(Least Recently Used and Least Frequently Used)缓存替换策略提出一种在HDFS客户端建立本地缓存的解决方案,从而降低Name Node负载。通过实验证明Hadoop应用可以从缓存中读取文件块信息,而不必请求Name Node,提高了数据访问效率,降低了Name Node负载。Hadoop distributed file system（ HDFS） is a distributed file system based on Master/Slave master-slave architecture. Name Node manages the metadata of all files and handles client-side access to files. As application data continues to increase,as Data Nodes continue to be added,the workload of Name Node maintaining file metadata and handling file operations such as read and delete of Hadoop applications increases. The performance of Name Node becomes the bottleneck of the entire HDFS. In this paper,we proposed an idea that some of the Name Node tasks should be submitted to the HDFS client so that each HDFS client became a Name Node on some functions. Based on the least recently used and least frequently used（ LRFU） cache replacement strategy,a solution to establish a local cache on the HDFS client was proposed to reduce the Name Node load. Experiments show that Hadoop applications read file block information from the cache without having to request Name Node,which improves data access efficiency and reduces Name Node load.

关 键 词：HDFS 主从架构 LRFU 客户端 缓存 

分 类 号：TP302.1[自动化与计算机技术—计算机系统结构]