大规模异构计算集群的双层作业调度系统  

Dual Layer Job Scheduling System for Large Scale Heterogeneous Computing Clusters

作　　者：孙震宇[1,2] 石京燕 孙功星[1] 杜然[1] 姜晓巍[1] 邹佳恒[1] 谭宏楠[1,2] SUN Zhenyu;SHI Jingyan;SUN Gongxing;DU Ran;JIANG Xiaowei;ZOU Jiaheng;TAN Hongnan(Institute of High Energy Physics,Chinese Academy of Sciences,Beijing 100049,China;School of Physical Sciences,University of Chinese Academy of Sciences,Beijing 100049,China)

机构地区：[1]中国科学院高能物理研究所,北京100049 [2]中国科学院大学物理科学学院,北京100049

出　　处：《计算机工程》2020年第1期187-195,共9页Computer Engineering

基　　金：国家自然科学基金(11475210);国家自然科学基金青年基金(11805225)

摘　　要：高能物理计算平台中的HTCondor和SLURM计算集群为多个高能物理实验提供数据处理服务,然而HTCondor并行作业调度效率较低、SLURM难以应对大量串行作业,且计算平台整体资源管理及调度策略过于简单。为满足高能物理计算集群高负荷运行的需求,在传统作业调度器上增加作业管理层,设计双层作业调度系统,通过高效调度串并行作业并兼顾实验组间资源的使用公平性,实现用户对作业的细粒度管理。测试结果表明,双层作业调度系统支持大批量高能物理作业的快速提交,并充分利用计算平台的总体资源,具有较好的作业调度性能。HTCondor and SLURM computing clusters in high-energy physics computing platforms provide data processing services for many high-energy physics experiments.However,HTCondor is not efficient in parallel job scheduling,and SLURM could not manage massive serial jobs.Also,the overall resource management and scheduling strategies of computing platforms are too simple.To meet the demands of high-energy physics computing clusters running with heavy duties,this paper designs a dual layer job scheduling system,which adds a job management layer on the existing job scheduler.The system is designed to efficiently schedule serial and parallel jobs,ensure fair use of resources between experiment groups,and enable users to implement fine-grained management of jobs.Test results show that the dual layer job scheduling system supports rapid submission of massive high-energy physics jobs,makes full use of resources of a computing platform,and has a high performance in job scheduling.

关 键 词：计算集群管理 作业调度器 高通量计算 高性能计算 高能物理计算 

分 类 号：TP319[自动化与计算机技术—计算机软件与理论]