基于Spark和小波分析的水上交通异常数据实时检测方法研究  

Research on real-time detection method of water traffic abnormal data based on Spark and wavelet analysis

作　　者：杨帆[1] 何正伟[1,2,3] 刘力荣 YANG Fan;HE Zheng-wei;LIU Li-rong(School of Navigation,Wuhan University of Technology,Wuhan 430063,China;Hubei Inland Shipping Technology Key Laboratory,Wuhan 430063,China;National Engineering Research Center for Water Transport Safety,Wuhan 430063,China)

机构地区：[1]武汉理工大学航运学院,湖北武汉430063 [2]内河航运技术湖北省重点实验室,湖北武汉430063 [3]国家水运安全工程技术研究中心,湖北武汉430063

出　　处：《陕西理工大学学报（自然科学版）》2019年第1期35-41,共7页Journal of Shaanxi University of Technology:Natural Science Edition

基　　金：武汉理工大学自主创新研究基金资助项目(185212008);中央高校基本科研业务费专项资金资助项目

摘　　要：针对不断增加的水路运输产生的水上交通数据量增长,导致的水上交通监测难度更大、处理时间更长,提出一种基于Spark的交通异常数据实时检测方法,通过对船舶自动识别系统(AIS)数据进行处理,对不同类型的交通数据进行分析并写入分布式文件系统HDFS中。然后通过小波分析的方法对AIS数据进行多层分解,去除高频噪声并对数据进行重构,找出AIS数据中的异常信息。通过对异常信息进行分析,结合Spark的数据处理结果,最终实时检测出交通异常数据。实验结果表明能够在短时间内对异常数据进行检测和分析,处理速度快,异常数据检测结果符合该水域的交通情况,检测方法能够为海事部门提供实时、稳定的监管服务。With the increasing amount of waterway transportation,the amount of data generated by water traffic is also increasing,which makes the monitoring of water traffic more difficult and more timeconsuming.In this paper,a real-time detection method of traffic abnormal data based on Spark is proposed.By analyzing the AIS data,different types of traffic data are analyzed and written into the distributed file system HDFS.Then,the AIS data is decomposed by wavelet analysis,the high frequency noise is removed and the data is reconstructed to find the abnormal information in AIS data.Through the analysis of abnormal information,combined with Spark data processing results,the final real-time traffic abnormal data is detected.The experimental results show that the abnormal traffic data can be detected and analyzed in a short time,the processing speed is fast and the abnormal data test result is consistent with the traffic situation of the waters.The detection method can provide the maritime department with real-time and stable regulatory services.

关 键 词：交通异常信息 大数据 船舶自动识别系统 小波分析 

分 类 号：TP274.2[自动化与计算机技术—检测技术与自动化装置]