基于HBase的证券交易数据实时采集系统的应用研究  

Application of Real-time Data Collection System for Securities Trading Based on HBase

作　　者：丁俊[1] 郑辉[1] Ding Jun;Zheng Hui(College of Computer and Art,Anhui Technical College of Industry and Economy,Hefei,Anhui 230051,China)

机构地区：[1]安徽工业经济职业技术学院计算机与艺术学院

出　　处：《黑龙江工业学院学报（综合版）》2019年第12期42-49,共8页Journal of Heilongjiang University of Technology(Comprehensive Edition)

基　　金：安徽省高校自然科学研究重点项目(KJ2018A0764)

摘　　要：随着大数据时代的到来,数据逐步上升为一种战略资源,在大数据产业链中,数据采集作为整个产业链的源头,直接影响着后续各个环节的发展。试提出一种数据采集方案,专门用来采集WEB网页中通过Ajax动态产生的JSON数据,同时利用Quartz和TimerTask双重定时机制,实现在指定的时点,在指定的周期内,按照指定的频率进行数据采集,以适应证券交易数据实时动态产生的应用场景,动态产生的实时JSON数据被采集后,系统再对其进行解析处理,并存储到分布式数据库HBase中,为后续的大数据实时统计分析和挖掘提供数据源。With the emergence of the big data,data has gradually taken as a strategic resource.In the big data industry chain,data collection has affected the development of subsequent links.This paper proposes a data collection scheme which is specially used to collect JSON data generated dynamically by Ajax in web pages.At the same time,the dual timing mechanism of Quartz and TimerTask is used to realize data collection at specified time points and in specified periods according to specified frequencies in order to adapt to the scenario which dynamically generate the data of securities trading.After being collected,the real-time JSON data generated dynamically will be parsed and processed by the system,and stored in the distributed database HBase,which will provide data source for the subsequent real-time statistical analysis and mining of large data.

关 键 词：数据采集 定时机制 JSON数据 动态 实时 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]