面向并发数据流处理的存储接入方法  

Storage access method for concurrent data stream processing

作　　者：胡汝博 陈庆奎[1] HU Rubo;CHEN Qingkui(School of Optical-Electrical and Computer Engineering,University of Shanghai for Science and Technology,Shanghai 200093,China)

机构地区：[1]上海理工大学光电信息与计算机工程学院,上海200093

出　　处：《智能计算机与应用》2024年第3期37-45,共9页Intelligent Computer and Applications

基　　金：国家自然科学基金(61572325);上海重点科技攻关项目(16DZ1203603,19DZ1208903);上海智能家居大规模物联共性技术工程中心项目(GCZX14014);上海市一流学科建设项目(XTKX2012);上海市重点项目(9DZ1208903)。

摘　　要：海量的并发数据流存储请求,会造成数据库瞬时压力过大和存储时延提高等问题,对此提出一种面向并发数据流处理的存储接入方法。首先构建Redis缓冲区队列,以接收上游数据处理节点的周期数据,使用自定义数据流协议解决数据流规范化通信;然后设计主动存储中间件,通过线程读写任务分配算法(TRWTA),优先处理高负载区队列的读写任务,利用动态线程池计算读写压力(RWP)来优化管理线程数量,达到提高并发数据流存储效率的目的。此外,在存储层中使用了Mysql和HBase对数据流进行分类存储,对HBase进行Rowkey设计和预分区。经过实验分析,所提存储接入方法在持续的并发数据流下,较传统存储方法有效降低了存储时延,保证了数据库稳定性。Massive concurrent data stream storage requests can cause excessive transient pressure on databases and increase storage latency.This paper proposes a storage access method for concurrent data stream processing.Firstly,a Redis buffer queue is constructed to receive periodic data from upstream data processing nodes,and a customized data flow protocol is used to solve data flow standardization communication.Then,an active storage middleware is designed to prioritize read and write tasks for queues in high load areas through a thread read and write task allocation algorithm(TRWTA).Dynamic thread pools are also used to calculate read and write pressure(RWP)to optimize the number of managed threads,achieving the purpose of improving the storage efficiency of concurrent data streams.In addition,MySQL and HBase are used in the storage layer to classify and store data streams,and HBase is designed for Rowkey and pre-partitioning.Through experimental analysis,the proposed storage access method effectively reduces storage latency and ensures database stability compared to traditional storage methods under continuous concurrent data streams.

关 键 词：并发数据流 主动存储中间件 读写任务分配 动态线程池 存储接入 

分 类 号：TP391[自动化与计算机技术—计算机应用技术] TP311[自动化与计算机技术—计算机科学与技术]