MapReduce模型在大规模数据并行挖掘中的应用  

The Application of MapReduce Model in Large-Scale Data Parallel Mining

作　　者：唐婧[1] 杜微[1] 周翼[1] TANG Jing;DU Wei;ZHOU Yi(Changde Vocational and Technical College,Changde 415000,China)

机构地区：[1]常德职业技术学院,湖南常德415000

出　　处：《智能物联技术》2024年第2期38-42,共5页Technology of Io T& AI

基　　金：2022年湖南省教育厅科学研究项目(22C0733);2022年常德市科技局创新指导性计划项目(2022ZD04)。

摘　　要：MapReduce并行编程模型通过定义良好的接口和运行支持库,能够自动并行执行大规模计算任务,隐藏底层实现细节,降低并行编程的难度。系统阐述MapReduce的基本工作原理及其工作流程,以TeraSort算法为例,针对其存在的问题,提出动态数据分区和数据压缩等优化建议。研究成果表明,优化后的TeraSort算法能够显著缩短数据处理时间,优化系统的吞吐量,并改善资源分配的均衡性。The MapReduce parallel programming model,through well-defined interfaces and runtime support libraries,can automatically execute large-scale computing tasks in parallel,hiding the underlying implementation details and reducing the complexity of parallel programming.This study systematically explains the basic working principles and workflow of MapReduce,using the TeraSort algorithm as an example,and proposes optimizations such as dynamic data partitioning and data compression to address existing issues.The research results show that the optimized TeraSort algorithm significantly reduces data processing time,enhances system throughput,and improves the balance of resource allocation.

关 键 词：MAPREDUCE 大规模数据 并行挖掘 TeraSort 

分 类 号：TP311.5[自动化与计算机技术—计算机软件与理论]