云原生数据湖服务平台的设计与实现  

Design and Implementation of a Cloud Native Data Lake Platform

作　　者：魏文定 鄂海红[1] 王曦[2] 宋美娜[1] 宿兴辉 WEI Wen-ding;E Hai-hong;WANG Xi;SONG Mei-na;SU Xing-hui(School of Computer Science(National Pilot Software Engineering School),Beijing University of Posts and Telecommunications,Beijing 100876,China;Institute of Information Engineering,Chinese Academy of Sciences,Beijing 100085,China;Lianyang Guorong(Beijing)Technology Co.,Ltd.,Beijing 100088,China)

机构地区：[1]北京邮电大学计算机学院(国家示范性软件学院),北京100876 [2]中国科学院信息工程研究所,北京100085 [3]联洋国融(北京)科技有限公司,北京100088

出　　处：《计算机技术与发展》2024年第2期17-22,共6页Computer Technology and Development

基　　金：国家自然科学基金(62176026);北京自然科学基金(M22009)。

摘　　要：云原生数据湖已经成为数据管理和分析领域的研究热点,相关技术和应用也得到了广泛的关注和探索。数据湖部署存在着成本高、组件之间兼容性差等问题,存算不分离制约着数据湖平台延展性,缺乏完备的数据入湖体系容易引起数据湖沼泽的形成,导致用户无法从中提取数据价值。该文设计并实现了云原生数据湖服务平台,平台以Kubernetes为底层构建云原生环境,结合容器技术将数据湖组件镜像化,同时设计数据湖存算分离方案来提高数据湖平台的可扩展性和可移植性,并配合监控、组装生产线将镜像容器化,实现数据湖上云操作。并建立用户入湖作业与云原生计算引擎之间的桥梁,对入湖信息进行预处理,提供多类型作业以满足多元化入湖场景,以统一catalog的方式将数据写入数据湖中。实际运行结果表明,该平台既提高了数据湖平台的灵活性和可靠性,又确保了元数据和数据资产的规范性存储。Cloud native data lakes have become a research hotspot in the field of data management and analytics,and related technologies and applications have received widespread attention and exploration.Data lake deployment suffers from high cost and poor compatibility between components,the lack of separation of storage and computation restricts the extensibility of the data lake platform,and the lack of a complete data entry system easily causes the formation of a data lake swamp,resulting in users being unable to extract data value from it.We design and implement a cloud native data lake service platform,which uses Kubernetes as the underlying layer to build a cloud native environment,and combines container technology to mirror data lake components.Meanwhile,the storage and computing separation scheme of the data lake is designed to improve the scalability and portability of the data lake platform,and the image is containerized with the monitoring and assembly production line to realize cloud operations on the data lake.The platform also establishes a bridge between the user’s entry operations and the cloud native computing engine,pre-processes the entry information,provides multiple types of operations to meet diverse entry scenarios,and writes data to the data lake in a unified catalog manner.The actual operation results show that the platform not only improves the flexibility and reliability of the data lake platform,but also ensures the normative storage of metadata and data assets.

关 键 词：云原生 数据湖 大数据 生产线 数据湖上云 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]