面向数据生产的跨计算引擎数据调度技术研究  

Research on Cross-computing Engine Data Scheduling Technology for Data Production

作　　者：雷鸣 李丹[2] LEI Ming;LI Dan(China Justice Big Data Research Institute,Beijing 100043,China;China Academy of Electronic and Information Technology,CETC,Beijing 100041,China)

机构地区：[1]中国司法大数据研究院有限公司,北京100043 [2]中国电子科学研究院,北京100041

出　　处：《中国电子科学研究院学报》2024年第6期570-577,共8页Journal of China Academy of Electronics and Information Technology

摘　　要：随着数据处理任务的日渐复杂,传统的数据计算引擎面临着存储管理和计算资源调度的挑战。本文深入探讨了如何提升跨计算引擎数据资源调度的性能和降低其复杂性。研究聚焦于数据生产任务,通过结合实际应用场景,重点研究了数据资源在不同计算引擎间的存储和调度优化技术,提出了超级调度器设计、多维度细粒度资源分配以及基于任务优先级的资源调度等技术方案。实现不同计算任务在多个计算引擎间的高效协作,从而最大化计算资源的利用率和处理速度,为解决大规模数据生产中的资源调度问题提供了有效的技术方案。With the increasing complexity of data processing tasks,traditional data computation engines face challenges in storage management and computational resource scheduling.This paper provides an in-depth discussion on how to improve the performance and reduce the complexity of cross-computing engine data resource scheduling.Focusing on data production tasks,the research focuses on storage and schedu-ling optimization techniques for data resources across computing engines by combining practical applica-tion scenarios,and proposes technical solutions such as super-scheduler design,multi-dimensional fine-grained resource allocation,and task priority-based resource scheduling.It realizes the efficient collabo-ration of different computing tasks among multiple computing engines,thus maximizing the utilization rate and processing speed of computing resources,and provides an effective technical solution for solving the resource scheduling problem in large-scale data production.

关 键 词：数据调度 跨计算引擎 数据生产 任务调度器 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论] TN929.5[自动化与计算机技术—计算机科学与技术]