面向外汇市场监测的分布式计算框架设计  

Design of distributed computing framework for foreign exchange market monitoring

作　　者：程文亮 王志宏 周虞 过弋[2,3,4] 赵俊锋 CHENG Wenliang;WANG Zhihong;ZHOU Yu;GUO Yi;ZHAO Junfeng(Research and Development 2 Department,China Foreign Exchange Trade System Information Technology(Shanghai)Company Limited,Shanghai 201203,China;School of Computer Science and Engineering,East China University of Science and Technology,Shanghai 200237,China;National Engineering Laboratory for Big Data Distribution and Exchange Technologies(Business Intelligence and Visualization Research Center),Shanghai 200237,China;Shanghai Engineering Research Center of Big Data and Internet Audience,Shanghai 200072,China)

机构地区：[1]中汇信息技术(上海)有限公司开发二部,上海201203 [2]华东理工大学信息科学与工程学院,上海200237 [3]大数据流通与交易技术国家工程实验室(商业智能与可视化研究中心),上海200237 [4]上海大数据与互联网受众工程技术研究中心,上海200072

出　　处：《计算机应用》2020年第1期173-180,共8页journal of Computer Applications

基　　金：国家重点研发计划项目(2018YFC0807105);上海市科学技术委员会科研计划项目(17DZ1101003,18511106602,18DZ2252300)~~

摘　　要：针对金融外汇市场监测指标计算复杂度高、完备性强、效率低等问题,基于Spark大数据架构提出了一种新的面向外汇市场监测的分布式计算框架。首先,对外汇市场监测的业务特性和现有技术框架进行了分析总结;然后,综合考虑了外汇单市场多指标和多市场多指标并行计算的业务特性;最后,基于Spark的有向无环图(DAG)作业调度机制和YARN的资源调度池隔离机制,分别提出了外汇市场级的有向无环图(M-DAG)模型和市场级资源分配策略--M-YARN。实验结果表明,所提面向外汇市场监测的分布式计算框架相对于传统技术框架在性能上提高了80%以上,可以有效保证大数据背景下外汇市场监测指标计算的完备性、精准性和时效性。In order to solve the index calculation problems of high complexity, strong completeness and low efficiency in the filed of financial foreign exchange market monitoring, a novel distributed computing framework for foreign exchange market monitoring based on Spark big data structure was proposed. Firstly, the business characteristics and existing technology framework for foreign exchange market monitoring were analyzed and summarized. Secondly, the foreign exchange business features of single-market multi-indicator and multi-market multi-indicator were considered. Finally, based on Spark’s Directed Acyclic Graph(DAG) job scheduling mechanism and resource scheduling pool isolation mechanism of YARN(Yet Another Recourse Negotiator), the Market-level DAG(M-DAG) model and the market-level resource allocation strategy named M-YARN(Market-level YARN) model were proposed, respectively. The experimental results show that, the performance of the proposed distributed computing framework for foreign exchange market monitoring improves the performance by more than 80% compared to the traditional technology framework, and can effectively guarantee the completeness, accuracy and timeliness of foreign exchange market monitoring indicator calculation under the background of big data.

关 键 词：外汇市场 市场监测 SPARK 有限无环图 资源分配 

分 类 号：TP399[自动化与计算机技术—计算机应用技术]