一种科学计算应用的快速I/O方法——双路并行I/O  

An I/O Performance Optimization for Scientific Computing Applications——Dual Channel Parallel I/O

作　　者：曹立强[1] 沈卫超[1] 夏芳[1] 

机构地区：[1]北京应用物理与计算数学研究所,北京100094

出　　处：《科研信息化技术与应用》2014年第5期3-9,共7页E-science Technology & Application

基　　金：国家高技术研究发展计划(863计划)"高效能计算机应用支撑软件框架体系研制"(2012AA01A309);国家自然科学基金重点项目"面向重大应用领域的高效能计算优化理论与技术"(61033009);面上项目"大规模几何建模和自适应笛卡尔网格生成并行算法研究"(61170310);国家重点基础研究发展计划(973计划)"适应于千万亿次科学计算的新型计算模式"(2011CB309702)

摘　　要：许多并行科学计算应用使用HDF5等数据格式管理计算数据与元数据。一般条件下,科学计算数据与元数据有不同的存储特征。数据存储规模较大,可以在并行文件系统中获得较高的I/O效率。元数据存储规模较小,难以发挥并行文件系统带宽。一旦数据与元数据混合存储,元数据I/O将较多的干扰数据I/O,降低总的I/O效率。针对数据与元数据的不同I/O特征,本文设计双路I/O系统,该系统优化元数据的存储位置。将元数据存储于容量较小、但是延迟较低的固态盘文件系统中,在降低元数据I/O延迟的同时提高数据I/O的连续性。实验表明,双路并行I/O可以提高数据读带宽3倍左右。Scientific data and metadata are saved in formatted file to apply in different environments and across platforms. However, there are different characters in stored data and metadata. Data are large in size and rarely accessed. Meta data are small in size but frequently accessed. Existed ifle formats aligned data and metadata in order. This is inefifcient in the cache management of ifle system while writing data and there will be long latency in dis-contiguous metadata readings. In this paper, we present a dual channel I/O format and library, which saves scientiifc data and metadata in different storage respectively. Data are saved in the traditional hard disk and metadata are saved in the low latency solid-state disk or memory. Evaluation shows dual channel I/O library is faster 3 times in reading than traditional I/O library.

关 键 词：科学数据管理 数据模型 数据格式 I/O性能优化 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]