SKA-MWA天文数据存储优化与高效预处理方法研究  

Research on Storage Optimization and Efficient Pre-Processing Methods for SKA-MWAAstronomical Data

作　　者：周晗 唐家宁 薛梦瑶 吴开超[1] 张波[1] ZHOU Han;TANG Jianing;XUE Mengyao;WU Kaichao;ZHANG Bo(Computer Network Information Center,Chinese Academy of Sciences,Beijing 100083,China;National Astronomical Observatories,Chinese Academy of Sciences,Beijing 100101,China)

机构地区：[1]中国科学院计算机网络信息中心,北京100083 [2]中国科学院国家天文台,北京100101

出　　处：《数据与计算发展前沿(中英文)》2025年第2期49-59,共11页Frontiers of Data & Computing

基　　金：国家自然科学基金(6217023073)。

摘　　要：【背景】默奇森宽场阵列(Murchison Widefield Array,MWA)是平方公里阵列(Square Kilometre Array,SKA)低频先导望远镜,广泛用于脉冲星等天文现象的研究,由于其数据传输读写规模大,数据处理存在耦合,导致其读写性能低,影响数据处理效率。【目的】为提高MWA数据处理效率,通过预处理优化存储布局,缓解数据处理的读写瓶颈。【方法】通过分析MWA数据特性及计算流程,提出纵向的数据布局策略。本地计算模式与流水线架构的结合,实现高效的数据预处理,完成数据布局调整。【结果】该方案优化了数据存取策略,引入打包、压缩使得文件数减少到1/40,数据量减少到70%,结合本地计算模式,降低共享存储I/O负载,可大大提升天文数据分析的效率。采用本地计算模式的数据预处理方案,数据预处理计算效率提升了3倍以上。【结论】本文提出的数据布局策略与预处理的优化方法,提升了SKA-MWA天文数据的存储性能和后续波束合成的计算效率,为天文计算提供了高质量数据支撑,该方法具有普适性,有广泛的应用前景。[Context]The Murchison Widefield Array(MWA)is a low-frequency precursor telescope for the Square Kilometre Array(SKA),which is widely used in the study of astronomical phenomena such as pulsars.Due to the large scale of data transmission and storage,coupled with challenges in data processing,the read-write performance is low,thereby affecting the efficiency of data processing.[Object]To enhance the data processing efficiency of the MWA,a pre-processing optimization of storage layout is proposed to alleviate the read-write bottlenecks.[Methods]By analyzing the data characteristics and computational workflows of the MWA,a vertical data layout strategy is introduced.This approach,combining local computation modes with a pipeline architecture,achieves efficient data pre-processing and layout adjustment.[Results]The proposed solution optimizes the data access strategy by incorporating packing and compression techniques that reduce the number of files by a factor of 40 and the data volume by 70%.With the local computation mode,the shared storage I/O load is reduced,significantly enhancing the efficiency of astronomical data analysis.The data pre-processing solution using local computation mode achieves threefold improvement in computational efficiency.[Conclusions]The data layout strategy and pre-processing optimization methods proposed in this study can significantly improve the storage performance of SKA-MWA astronomical data and the computational efficiency of subsequent beamforming.This approach provides high-quality data support for astronomical computations and is universally applicable with broad prospects for future application.

关 键 词：数据预处理 分布式并行 存储优化 本地存储 

分 类 号：P111[天文地球—天文学]