面向航空制造业的海量数据处理研究  

Research on Mass Data Processing Oriented to Aircraft Manufacturing Industry

作　　者：吴恒[1] 王东勃[1] 

机构地区：[1]西北工业大学机电学院,陕西西安710072

出　　处：《机械设计与制造工程》2013年第4期28-31,共4页Machine Design and Manufacturing Engineering

摘　　要：为了向航空制造业的海量数据处理提供一种新颖的模式,以Hadoop开源软件平台为架构,介绍了HDFS分布式存储系统和Map-Reduce编程模式,分析了航空制造业海量数据处理需求,提出了一种应用于航空制造业的海量数据处理模型,即将数据格式划分为若干个主要字段,按照该数据格式在分片中对数据进行深度优先挖掘,将提取出的数据以键值对的形式并按照一定的存储格式存储于HDFS中,利用Map-Reduce并行算法对存储的数据进行排序和分区处理。最后提出了基于K-NN的并行化数据挖掘算法,且分析了海量数据处理模型的扩展性、实时性和快速处理等性能。In order to provide a new model of mass data processing for aircraft manufacturing industry, it proposes a model of mass data processing for aircraft manufacturing industry based on Hadoop software platform, intro- duces the HDFS and the Map - Reduce programming model, analyzes the demand of mass data processing for aircraft manufacturing industry, introduces a kind of data processing model. This system can divide the data for- mat into several major fields, mine the data from shaming deeply and firstly according to the data format, store the extracted data in HDFS in the format of key - value and the defined storage format, sort and partition the storage data in the method of Map - Reduce. Lastly, it illustrates a parallel data mining algorithm based K - NN, and analyzes the expansibility, instantaneity and quick processing of this kind of mass data processing model.

关 键 词：航空制造业 海量数据 HADOOP 数据处理模型 键值对 算法 

分 类 号：TP399[自动化与计算机技术—计算机应用技术]