云计算环境下的大数据存储与处理技术研究  

Research on Big Data Storage and Processing Technology in Cloud Computing Environment

作　　者：许教美 XU Jiaomei(Guangzhou Pearl-River Vocational College of Technology,Guangzhou Guangdong 511300,China)

机构地区：[1]广州珠江职业技术学院,广东广州511300

出　　处：《信息与电脑》2024年第18期185-187,共3页Information & Computer

摘　　要：随着数据规模的不断扩大,云计算环境下的大数据存储与处理技术面临着前所未有的挑战。为了应对这些挑战,本研究以Brotli压缩算法为例,探讨了高效数据压缩算法和机器学习技术在优化大数据处理中的应用。在实验中,引入了Gzip和Snappy压缩算法,与Brotli压缩算法进行了性能对照,并将压缩算法与Spark MLlib库结合,进行了数据预处理、特征提取、分类和聚类分析等操作,以测试压缩算法对数据存储与处理的影响。实验结果表明,Brotli算法在压缩比和数据处理准确性方面表现最佳,Snappy算法则在数据传输速度方面具有明显优势。通过这些优化措施,本研究实现了在云计算环境下大数据存储与处理效率和准确性的显著提高。With the continuous expansion of data scale,big data storage and processing technologies in cloud computing environment are facing unprecedented challenges.In order to cope with these challenges,this study takes Brotli compression algorithm as an example and explores the application of efficient data compression algorithms and machine learning techniques in optimizing big data processing.In the experiments,Gzip and Snappy compression algorithms were introduced to compare the performance with the Brotli compression algorithm,and the compression algorithm was combined with the Spark MLlib library to carry out the operations of data preprocessing,feature extraction,classification,and clustering analysis to test the impact of the compression algorithms on the data storage and processing.The experimental results show that the Brotli algorithm performs best in terms of compression ratio and data processing accuracy,while the Snappy algorithm has a significant advantage in terms of data transfer speed.Through these optimization measures,this study achieves a significant improvement in the efficiency and accuracy of big data storage and processing in the cloud computing environment.

关 键 词：云计算 大数据 压缩算法 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]