基于Apache Arrow的大数据共享技术研究  

Research on Big Data Sharing Technology Based on Apache Arrow

作　　者：顾锡华 GU Xihua(State Grid Jiangsu Electric Power Co.,Ltd.,Wuxi Power Supply Branch,Wuxi,Jiangsu 214061,China)

机构地区：[1]国网江苏省电力有限公司无锡供电分公司,江苏无锡214061

出　　处：《移动信息》2024年第4期207-209,共3页MOBILE INFORMATION

摘　　要：Apache Arrow是一种列式内存数据交换格式,在大数据领域得到了广泛的关注和应用。文中针对大数据共享领域中存在的数据传输效率低下、数据类型转换困难等问题,提出了一种基于Apache Arrow的大数据共享技术方案,以提高数据传输效率,降低数据类型转换的复杂性。实验证明,该技术能显著提高数据传输效率,在不同数据类型转换场景下,均能取得较好的性能表现。Apache Arrow is a columnar in-memory data exchange format that has received extensive attention and application in the field of big data.Aiming at the problems of low data transmission efficiency and difficulty in data type conversion in the field of big data sharing,this paper proposes a big data sharing technology scheme based on Apache Arrow to improve data transmission efficiency and reduce the complexity of data type conversion.Experiments verify that this technology can significantly improve data transmission efficiency and achieve good performance in different data type conversion scenarios.

关 键 词：大数据 数据交换 数据共享 Apache Arrow 

分 类 号：TP274.2[自动化与计算机技术—检测技术与自动化装置]