Offline traffic analysis system based on Hadoop  

作　　者：QIAO Yuan-yuan LEI Zhen-ming YUAN Lun GUO Min-jie 

机构地区：[1]Beijing Key Laboratory of Network System Architecture and Convergence,Beijing University of Posts and Telecommunications [2]Produce Ads,Amazon Joyo Co. Ltd

出　　处：《The Journal of China Universities of Posts and Telecommunications》2013年第5期97-103,共7页中国邮电高校学报（英文版）

基　　金：supported by the Important National Science & Technology Specific Projects (2012ZX03002008);the National Natural Science Foundation of China (61072061);The Fundamental Research Funds for the Central Universities (2012RC0121)

摘　　要：Offiine network traffic analysis is very important for an in-depth study upon the understanding of network conditions and characteristics, such as user behavior and abnormal traffic. With the rapid growth of the amount of information on the Intemet, the traditional stand-alone analysis tools face great challenges in storage capacity and computing efficiency, but which is the advantages for Hadoop cluster. In this paper, we designed an offiine traffic analysis system based on Hadoop （OTASH）, and proposed a MapReduce-based algorithm for TopN user statistics. In addition, we studied the computing performance and failure tolerance in OTASH. From the experiments we drew the conclusion that OTASH is suitable for handling large amounts of flow data, and are competent to calculate in the case of single node failure.Offiine network traffic analysis is very important for an in-depth study upon the understanding of network conditions and characteristics, such as user behavior and abnormal traffic. With the rapid growth of the amount of information on the Intemet, the traditional stand-alone analysis tools face great challenges in storage capacity and computing efficiency, but which is the advantages for Hadoop cluster. In this paper, we designed an offiine traffic analysis system based on Hadoop （OTASH）, and proposed a MapReduce-based algorithm for TopN user statistics. In addition, we studied the computing performance and failure tolerance in OTASH. From the experiments we drew the conclusion that OTASH is suitable for handling large amounts of flow data, and are competent to calculate in the case of single node failure.

关 键 词：MAPREDUCE HADOOP cloud computing traffic analysis 

分 类 号：TP393.06[自动化与计算机技术—计算机应用技术]