基于Apache Spark的地震观测数据噪声功率谱计算  

Noise Power Spectrum Calculation Method of Seismic Data Based on Apache Spark

作　　者：郭凯 黎建辉[1] 温亮明[1,2] 韩振华 GUO Kai;LI Jian-Hui;WEN Liang-Ming;HAN Zhen-Hua(Computer Network Information Center,Chinese Academy of Sciences,Beijing 100190,China;University of Chinese Academy of Sciences,Beijing 100049,China;China Seismic Network Center,Beijing 100045,China;Taiyuan University of Technology,Taiyuan 030024,China)

机构地区：[1]中国科学院计算机网络信息中心,北京100190 [2]中国科学院大学,北京100049 [3]中国地震台网中心,北京100045 [4]太原理工大学,太原030024

出　　处：《计算机系统应用》2021年第8期126-132,共7页Computer Systems & Applications

基　　金：国家重点研发计划(2018YFC1504500);中国地震局监测、预报、科研三结合课题(3JH-20200207)。

摘　　要：为解决单机环境下海量地震观测数据计算和分析效率低下的问题,提出一种基于分布式架构的地震观测数据的存储、计算和分析处理方法,选择噪声功率谱复杂计算过程的应用场景进行实现.基于Hadoop在海量数据处理上的性能优势,在分布式文件存储系统HDFS上进行地震观测数据的存储和调度,研究测震数据噪声功率谱的质量评估方法在Spark分布式计算架构上的实现,采用弹性数据集Spark RDD将计算任务自动分配到计算节点,解析存储在HDFS中的测震波形数据,计算结果采用RowKey方式放入分布式数据库HBase中,实现了长周期地震噪声功率谱结果的存储和提取.计算结果表明,基于Spark分布式架构的该方法可以支撑TB级海量数据的处理,并且具有较高的处理效率,可应用于海量地震观测数据的分析计算.To solve the problem of inefficient calculation and analysis of massive seismic data in a single machine environment,we propose a distributed architecture based method for storage,calculation,and analysis of seismic data and select the complex calculation process of a noise power spectrum as the application scenario for implementation.In light of Hadoop’s performance advantage in massive data processing,the storage and scheduling of seismic data are carried out on the Hadoop Distributed File System(HDFS).The implementation of the quality evaluation method for the noise power spectrum of seismic data in Spark distributed computing architecture is studied.The elastic dataset Spark RDD is used to automatically allocate the tasks to the computing nodes,and the seismic waveform data stored in HDFS is analyzed.In addition,the calculation results are input into the distributed database HBase in the RowKey mode,realizing the storage and extraction of the power spectra of long-period seismic noise.The calculation results show that the method based on Spark distributed architecture can support the efficient processing of massive data at the TB level in volume,which can be applied to the analysis and calculation of massive seismic data.

关 键 词：地震观测数据 噪声功率谱 SPARK HADOOP 分布式 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论] P315.63[自动化与计算机技术—计算机科学与技术]