端到端的科学数据跨中心工作流分析框架  

End-to-End Workflow Framework for Cross-Center Scientific Data Analysis

作　　者：朱小杰[1] 王华进 沈志宏[1] 郭学兵[2] 董文 ZHU Xiaojie;WANG Huajin;SHEN Zhihong;GUO Xuebing;DONG Wen(Computer Network Information Center,Chinese Academy of Sciences,Beijing 100083,China 2.Institute of Geographic Sciences and Natural Resources Research,Chinese Academy of Sciences,China)

机构地区：[1]中国科学院计算机网络信息中心,北京100083 [2]中国科学院地理科学与资源研究所,北京100101

出　　处：《数据与计算发展前沿》2023年第1期15-27,共13页Frontiers of Data & Computing

基　　金：国家重点研发计划“面向国家科学数据中心的基础软件栈及系统”(No.2021YFF0704200);中国科学院“十四五”网信专项工程建设项目“科学大数据工程(三期)”(No.CAS-WX2022GC-02)。

摘　　要：【目的】大数据与人工智能技术的快速发展催生了科研范式变革,新科研范式普遍要求对不同领域的科学数据资源进行协同分析,任务类型多样,分析流程横跨不同科学数据中心。【应用背景】现有工作流分析框架因在分析流程表达能力、异构计算框架整合能力、跨中心作业调度能力上的不足,难以支撑端到端的科学数据跨中心工作流分析需求。【方法】本文提出了可进行端到端科学数据跨中心工作流分析的软件框架,支持跨中心异构工作流构建、跨框架数据透明传递、跨中心作业优化调度。【结果】基于国家生态科学数据中心“草地地上生物量跨台站在线处理与质量控制”场景,对科学数据跨中心工作流分析框架进行了功能和性能验证,验证了该框架的先进性和可行性。[Objective]The rapid development of big data and artificial intelligence technology has led to the transformation of research paradigms.New paradigms generally require collaborative analysis.Task types are complex and the analysis process spans different scientific data centers.[Application background]Existing process-based analysis frameworks are difficult to support end-to-end crosscenter scientific data analysis requirements due to the lack of the capabilities of analysis process expression,heterogeneous computing framework integration,and cross-center job scheduling.[Methods]A software framework for end-to-end cross-center analysis of scientific data is proposed,which supports cross-center heterogeneous workflow construction,cross-framework data transparent transfer,and cross-center job optimization scheduling.[Results]The function and performance of the proposed framework are verified based on the scenario of"cross-station online processing and quality control of aboveground grass biomass"in the National Ecosystem Science Data Center,which verifies the advancement and feasibility of the framework.

关 键 词：科研范式 工作流分析 科学数据中心 跨中心计算 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]