基于StarRocks的实时物联网数据处理系统  

Real Time IoT Data Processing System Based on StarRocks

作　　者：董一舟 潘伟华 张楠 孟壮 DONG Yizhou;PAN Weihua;ZHANG Nan;MENG Zhuang(China Electronics Technology Digital Technology Group Co.,Ltd.,Shanghai 201808,China;Yingtan Big Data Center,Yingtan 335000,China)

机构地区：[1]中电科数字科技(集团)有限公司,上海201808 [2]鹰潭市大数据中心,江西鹰潭335000

出　　处：《计算机与现代化》2025年第1期15-19,共5页Computer and Modernization

基　　金：国家重点研发计划(2020YFB2104200);江西省03专项及5G项目(20212ABC03A11)。

摘　　要：随着物联网技术的普及和应用,大量的实时数据需要被处理和分析,因为物联数据的海量性、实时性特点,传统数据库无法满足其数据存储规模和数据处理效率的要求。本文提出一种基于StarRocks的分布式实时物联网数据处理系统。该系统利用StarRocks的分布式架构构建底层数据存储,通过引入消息队列和数据合并批量提交技术,保证数据的快速写入;同时通过存储策略优化、索引优化、物化视图技术,实现对大规模实时数据的快速处理和查询;系统强大的数据压缩能力也有效节省了数据存储空间。该框架在数据存储规模上支持横向扩展,提高了可用性和健壮性。通过实验分析,该系统在数据写入、数据查询、数据压缩方面较传统分布式数据库具有明显优势。With the popularization and application of IoT technology,a large amount of real-time data needs to be processed and analyzed.Due to the massive and real-time characteristics of IoT data,traditional databases cannot meet the requirements of data storage scale and data processing efficiency,this article proposes a distributed real-time IoT data processing system based on StarRocks.The system utilizes the distributed architecture of StarRocks to build underlying data storage.By introducing message queues and data merging batch submission technology,it ensures fast data writing.At the same time,through storage strategies optimization,index optimization,and materialized view technology,rapid processing and querying of large-scale real-time data have been achieved.The powerful data compression capability of the system also effectively saves data storage space.This framework supports horizontal scaling in data storage scale,improving availability and robustness.Through experimental analysis,the system has significant advantages over traditional distributed databases in terms of data writing,data querying,and data compression.

关 键 词：StarRocks 实时数据处理 分布式系统 数据压缩 查询优化 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]