支持异构集群并行的高能物理数据处理系统  

High Energy Physics Data Processing System with Parallel Heterogeneous Clusters

作　　者：霍菁[1,2] 雷晓凤[1,2] 李强[1,2] 孙功星[1] 

机构地区：[1]中国科学院高能物理研究所,北京100049 [2]中国科学院大学,北京100049

出　　处：《计算机工程》2015年第1期1-5,共5页Computer Engineering

基　　金：国家自然科学基金资助项目(11375223;11375221);国家自然科学基金A3前瞻计划基金资助项目(61161140454)

摘　　要：传统集群计算系统无法充分利用本地磁盘的存储资源和I/O,大量网络I/O成为系统瓶颈,导致资源利用率降低,并造成高昂的存储和网络成本。使用Hadoop处理分析作业可有效利用本地磁盘存储和I/O资源,而集群资源统一管理工具Mesos则使用轻量化的设计和高效的通信机制,能在不同计算集群之间动态共享集群资源。为此,分析高能物理数据处理的特点,利用Mesos构建异构集群间资源共享的高能物理实验数据处理系统,实现Torque/Maui和Hadoop集群的集成。测试结果表明,该系统能够在集群间动态分配集群资源,并利用本地存储和磁盘I/O显著降低网络I/O,提高集群资源利用率。The traditional cluster computing system can not make best of the local disks and disk I/ O resources,therefore the network becomes the bottleneck of the whole system. And this is the reason of low utilization of the cluster resources and high cost on data storage and network equipment. Using Hadoop to process analysis can significantly reduce the pressure on network I/ O by using the local disks as a distributed file system. Mesos is a cluster resource manager with light-weight design and efficient communication mechanisms that can dynamically share resources among clusters. This paper introduces the features of High Energy Physics（HEP）,data processing,presents a new HEP data processing system by using Mesos to provide dynamic resource sharing among clusters,and implements integration of Toruqe/ Maui and Hadoop which can avoid the disadvantages. The test result shows that the new system can dynamic distribute the cluster resource,and reduce the network I/ O,improve the resource utilization.

关 键 词：高能物理 集群资源管理 资源共享 Mesos工具 HADOOP平台 Torque/Maui系统 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]