ONSET数据流水线  

A Data Pipeline for Optical and Near-infrared Solar Eruption Tracer

作　　者：王新华 陈东[1] 邓涛 代红兵[3] 向永源[1] Wang Xinhua;Chen Dong;Deng Tao;Dai Hongbing;Xiang Yongyuan(Yunnan Observatories,Chinese Academy of Sciences,Kunming 650216,China;University of Chinese Academy of Sciences,Beijing 100049,China;School of Information,Yunnan University,Kunming 650504,China)

机构地区：[1]中国科学院云南天文台,云南昆明650216 [2]中国科学院大学,北京100049 [3]云南大学信息学院,云南昆明650504

出　　处：《天文研究与技术》2022年第1期86-94,共9页Astronomical Research & Technology

基　　金：国家自然科学基金(U1831210,11973088)资助.

摘　　要：随着天文大科学设备的投入使用,传统的开发模式面临程序重复开发,环境依赖冲突等问题。另外,集群是一个高度耦合的计算资源,严重的环境冲突可能导致整个集群不可用。为了解决这个问题,采用微服务的概念开发新的流水线框架,这种框架可以实现短期内开发和部署新的流水线。介绍了通过这种框架开发的ONSET数据流水线,为了实现准实时数据处理,采用MPI和GPU技术对核心程序做了优化,并对最后的性能做了评估。结果表明,这种开发模式可以在短期内搭建满足需求的流水线,这种开发模式对未来多波段多终端的天文数据处理有借鉴意义。With the advent of large astronomical equipments,the traditional development model for data reduction faces problems such as redundancy of programs and conflicting environmental dependencies;Besides as a cluster is a highly coupled computing resource,serious environmental conflicts can lead to the unavailability of the entire cluster.To address this problem,we have developed a new pipeline framework using the concept of microservices.This paper presents the ONSET(Optical and Near-infrared Solar Eruption Tracer)data pipeline developed through this framework.To achieve near real-time data processing,we optimize the core program using MPI and GPU technologies and evaluate the final performance.The results show that this development model can be built in a short time to meet the requirements of the pipeline,and we believe that this development model has implications for future multi-band and multi-terminal astronomical data processing.

关 键 词：ONSET 数据流水线 容器 GPU 

分 类 号：P182.21[天文地球—天文学] TP751.2[自动化与计算机技术—检测技术与自动化装置]