基于Alluxio的新闻文本存储优化方法  

An optimization method of news text storage based on Alluxio

作　　者：桂静 王永滨[1] GUI Jing;WANG Yongbin(State Key Laboratory of Media Convergence and Communication,Communication University of China,Beijing100024,China;School of Computer and Cyber Sciences,Communication University of China,Bei‐jing100024,China)

机构地区：[1]中国传媒大学媒体融合与传播国家重点实验室,北京100024 [2]中国传媒大学计算机与网络空间安全学院,北京100024

出　　处：《中国传媒大学学报（自然科学版）》2023年第6期12-18,共7页Journal of Communication University of China：Science and Technology

摘　　要：新闻数据的指数级增长对新闻的高效存储和管理提出了重大挑战。为解决存储与计算分离模式下新闻文本的存储与读访问瓶颈问题,提出了一种基于数据编排平台Alluxio的新闻文本存储优化方法。首先,利用Alluxio对不同存储系统中的新闻文本进行缓存并提供统一命名空间,加速了计算应用的数据访问。其次,为了解决新闻文本存储在远程场景下所面临的读访问性能瓶颈问题,对新闻文本进行了基于目录聚合的合并,并利用最小完美哈希算法对新闻文本元数据构建索引,实现了新闻文本的快速检索。The exponential growth of news data has presented a significant challenge to efficiently storing and managing news.To address the bottleneck issues of news text storage and access in the separation mode of storage and computation,we proposed an optimization method for news text storage based on Alluxio,a data orchestration platform.Firstly,Alluxio was utilized to cache news texts across various storage systems and provided a unified namespace,enhancing data access for computational applications.Secondly,to overcome performance limitations in reading and accessing remotely stored news text,we incorporated directory aggregation for merging the news text and emploied the Minimum Perfect Hash algorithm to index metadata associated with the news text.This enables rapid retrieval of relevant information from the stored news texts.

关 键 词：分布式系统 新闻数据 小文件存储 缓存 Alluxio 

分 类 号：TP31[自动化与计算机技术—计算机软件与理论]