面向股票交易分析场景的流式大数据系统测试框架  

System Test Framework of Stream Data for Stock Trading Analysis Scenario

作　　者：史凌云 郑莹莹[2,3] 谭励 许利杰[2,4] 王伟 魏峻[2,3,4] SHI Ling-Yun;ZHENG Ying-Ying;TAN Li;XU Li-Jie;WANG Wei;WEI Jun(School of Computer and Information Engineering,Beijing Technology and Business University,Beijing 100048,China;Institute of Software,Chinese Academy of Sciences,Beijing 100190,China;University of Chinese Academy of Sciences,Beijing 100049,China;State Key Laboratory of Computer Science,Beijing 100190,China)

机构地区：[1]北京工商大学计算机与信息工程学院,北京100048 [2]中国科学院软件研究所,北京100190 [3]中国科学院大学,北京100049 [4]计算机科学国家重点实验室,北京100190

出　　处：《计算机系统应用》2020年第4期76-83,共8页Computer Systems & Applications

基　　金：北京市自然科学基金(4172013);北京市自然科学基金-海淀原始创新联合基金(L182007);国家自然科学基金(61802377,61702020)及其配套项目(PXM2018_014213_000033);国家重点研发计划(2016YFD0401104)。

摘　　要：分布式集群环境使得数据实时计算更为复杂,流式大数据处理系统的正确性难以保障.现有的大数据基准测试框架可以测试流式大数据处理系统的性能表现,但是普遍存在应用场景设计简单、评价指标不充分等不足.针对这一挑战,本文构造了一个面向股票交易场景的流式大数据基准测试框架,通过生成股票高频交易数据,测试系统在高流速场景下的延迟、吞吐量、GC时间、CPU资源等的性能表现.同时,通过横向测试验证流式大数据系统的扩展性.本文以Apache Spark Streaming为待测系统进行测试,实验结果表明,高流速场景下出现延迟增加、GC时间提高等性能下降问题,原因是系统输入速率的提高及并行度的增加.Distributed cluster environment makes real-time data computation more complex, and the correctness of stream large data processing system is difficult to guarantee. The existing large data benchmarking framework can test the performance of stream large data processing system, but there are many shortcomings such as simple application scenario design and inadequate evaluation index. To address this challenge, this study constructs a stream large data benchmarking framework for stock trading scenarios, generates high-frequency stock trading data through a flow-based data generator, and tests the performance of the system in high-speed scenarios in terms of delay, throughput, GC time, CPU resources, and so on. At the same time, the scalability of large data streaming system is verified by horizontal test. In this study, Apache Spark Streaming is used as the test system to test. The experimental results show that the performance degradation problems such as delay increase and GC time increase occur in high-speed scenarios because of the increase of input rate and parallelism of the system.

关 键 词：流式大数据处理 系统性能 基准测试 APACHE SPARK STREAMING 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论] F830.91[自动化与计算机技术—计算机科学与技术]