一种基于自动分区的海量科学数据计算框架  

A computing framework for massive scientific data based on auto-partitioning algorithm

作　　者：田杨 晏海华[1] TIAN Yang;YAN Haihua(School of Computer Science and Engineering,Beihang University,Beijing 100083,China)

机构地区：[1]北京航空航天大学计算机学院,北京100083

出　　处：《北京航空航天大学学报》2022年第6期1004-1012,共9页Journal of Beijing University of Aeronautics and Astronautics

摘　　要：在科学研究领域,存储容量、处理效率和分析精度并不能适应科学数据的指数级增长速度。通过对科学数据结构与标准的研究,提出了一个海量科学数据计算框架BSDF。提出了一种基于模型驱动的统一数据接口,实现对异构科学数据的无差别访问;提出了一种基于元数据的自动分区算法,通过参数预取与超平面维度计算确定任务颗粒度。实验结果表明:与H5Spark科学数据计算框架的基于9项基准测试的性能相比,BSDF计算框架提升了39%~68%;在特定领域PKTM的算法优化上,BSDF达到了41.62倍的加速比。In the scientific research field,storage capacity,processing efficiency and analysis accuracy cannot keep pace with the exponential growth rate of scientific data.Thus,a massive scientific data calculation framework named BSDF is proposed based on scientific data structure and standards.A unified data interface based on model-driving is integrated to implement indiscriminate access to heterogeneous scientific data.Then an auto-partitioning algorithm based on scientific metadata is proposed,which determines task granularities through parameter prefetching and hyperplane dimension calculation.Experimental results show that compared with the performance of the H5 Spark framework,that of the BSDF is increased by 39%-68% in nine benchmark tests.In the optimization of the domain-specific PKTM algorithm,a speedup ratio is increased by 41.62 times.

关 键 词：科学数据 模型驱动 分区算法 叠前时间偏移 软件工程 SPARK 

分 类 号：V221.3[航空宇航科学与技术—飞行器设计] TB553[理学—物理]