面向大数据异构系统的神威并行存储系统  

Sunway parallel storage system for big data heterogeneous system

作　　者：何晓斌 蒋金虎 HE Xiaobin;JIANG Jinhu(National Parallel Computing Engineering Technology Research Center,Beijing 100080,China;School of Computer Science and Technology,Fudan University,Shanghai 200433,China)

机构地区：[1]国家并行计算工程技术研究中心,北京100080 [2]复旦大学计算机科学技术学院,上海200433

出　　处：《大数据》2020年第4期30-39,共10页Big Data Research

摘　　要：随着大数据应用和传统高性能计算应用的融合以及异构计算的引入,传统面向高性能计算的并行存储系统面临着异构计算I/O支持差、性能干扰和效率低等问题。通过在系统架构引入多层次存储架构、设计缓存映射机制来减轻I/O负载。在转发服务层,调整I/O转发策略,均衡I/O负载。在后端存储层,对系统高可用功能进行调整,解决大数据I/O访问模式与原有高可用措施的冲突。经过优化设计和完善后的并行存储系统更好地适应了异构众核架构,使得某些应用获得了10倍以上的I/O性能提升。With the integration of big data applications and traditional high-performance computing applications and the introduction of heterogeneous computing,the traditional parallel storage system for high-performance computing faces the problems of poor I/O support,performance interference,and low efficiency.By introducing multi-level storage architecture into the system architecture,the cache mapping mechanism was designed to reduce the I/O load.The I/O forwarding strategy was adjusted in the forwarding service layer to balance the I/O load.In the back-end storage layer,the high availability function of the system was adjusted to solve the conflict between the big data I/O access mode and the original high availability functions.After optimized design and improvement,the parallel storage system can better adapt to the heterogeneous multi-core architecture,making some applications get more than 10 times of I/O performance improvement.

关 键 词：大数据 高性能计算 神威·太湖之光 异构 并行存储 

分 类 号：TP316[自动化与计算机技术—计算机软件与理论]