一种基于LSM树的键值存储系统性能优化方法  

A Performance Optimization Method for Key-Value Store Based on LSM-tree

作　　者：王海涛[1,2] 李战怀 张晓[1,2] 赵晓南 Wang Haitao;Li Zhanhuai;Zhang Xiao;Zhao Xiaonan(School of Computer Science, Northwestern Polytechnical University, Xi an 710129;Key Laboratory of Big Data Storage and Management (Northwestern Polytechnical University), Ministry of Industry and Information Technology, Xi an 710129)

机构地区：[1]西北工业大学计算机学院,西安710129 [2]大数据存储与管理工业和信息化部重点实验室(西北工业大学),西安710129

出　　处：《计算机研究与发展》2019年第8期1792-1802,共11页Journal of Computer Research and Development

基　　金：国家重点研发计划项目(2018YFB1004401);国家自然科学基金青年科学基金项目(61502392);国家自然基金面上项目(61472323)~~

摘　　要：目前,键值(key-value, KV)存储系统在众多数据密集型的应用系统中发挥着关键作用,例如页面索引、电子商务以及云存储系统等.在各种键值存储系统中,基于日志结构合并(log-structured merge, LSM)树的KV存储系统获得了广泛的应用.主要原因是基于LSM树的KV存储系统能够将随机写操作转化为顺序写操作,从而提升数据写性能.然而,这些存储系统也存在一些严重的性能问题.一方面,KV存储系统利用预写日志机制来保证写入数据的原子性和安全性,以便在系统发生故障时进行恢复,造成了数据的写放大.同时,日志的频繁更新也引入了严重的元数据负载,导致了额外的性能开销.另一方面,KV存储系统通常利用通用文件系统存储数据,而通用文件系统中许多KV存储系统不需要的功能和属性也会造成一定的性能开销.为了减小这些开销、提升KV存储系统写入性能,提出了RocksFS,一个针对基于LSM树的KV存储系统优化的文件系统.针对KV存储系统的负载模式简化文件系统结构,去除通用文件系统引入的负载,同时优化预写日志的存储结构和更新流程以减小其更新负载,提升写入性能.最后在普通硬盘和固态硬盘上,基于流行的KV存储系统RocksDB验证了该方法的有效性,结果显示相对于通用文件系统,RocksFS能够将小块KV数据写入和更新性能提高约8倍.Nowadays, persistent key-value (KV) stores play a critical role in a variety of modern data-intensive applications, such as Web indexing, e-commerce, and cloud data storage systems, etc. KV stores that are based on log-structured merge tree (LSM-tree) have attracted growing attention because of their ability to eliminate random writes and maintain acceptable read performance. However, they also suffer from some performance issues. On one hand, they need to leverage write-ahead log (WAL) files to guarantee the atomicity and safety of write operations to enable recovery in case of a crash. This will result in severe write amplification and metadata overhead because of frequent WAL file update, leading to performance degradation. On the other hand, these KV stores usually use a conventional local filesystem to store KV data, which can harm the performance due to unnecessary operations in the filesystem. In this paper, we present RocksFS, an optimized filesystem for KV stores based on LSM-tree. We simplify the filesystem to remove unnecessary functions and attributes to reduce filesystem overhead and redesign the format and I O path of WAL file to decrease metadata overhead. We compare RocksFS with conventional filesystems in the environment of RocksDB, a popular LSM-tree-based KV store. The experimental results demonstrate that RocksFS can observably improve the small key-value data write performance of RocksDB by 8x at most compared with traditional filesystems on both hard disk drive and solid state disk.

关 键 词：LSM树 键值存储 文件系统 预写日志 写性能 

分 类 号：TP315[自动化与计算机技术—计算机软件与理论]