基于Python的并行编程技术在批量气象规范报表入库处理中的应用  

An Application of Python-Based Parallel Programming in Batch Meteorological Excel Sheets' Processing and Loading into Database

作　　者：郭学兵[1] GUO Xue-bing(Key Laboratory of Ecosystem Network Observation and Modeling,Institute of Geographic Sciences and Natural Resources Research,CAS,Beijing 100101)

机构地区：[1]中国科学院地理科学与资源研究所生态系统网络观测与模拟重点实验室,北京100101

出　　处：《软件》2018年第7期24-29,共6页Software

基　　金：中国科学院战略性先导科技专项(A类)(XDA19020301)资助

摘　　要：为满足中国生态系统研究网络(英文全称Chinese Ecosystem Research Network,简称CERN)对大批量气象规范报表(Excel格式)进行高效快速转换并载入数据库的需求,作者在对当前流行的并行编程方法和技术进行对比研究的基础上,提出了利用Python的多进程编程技术实现报表处理的方案,建立了CERN气象规范报表的抽象数据转换模型,并编写了多进程并行处理程序,实现了将大批量气象报表的快速转换载入Oracle数据库的功能。同时,作者对多进程编程的不同参数设置情景下的转换效率进行了测试、比较和分析。实验结果显示,利用Python并行编程技术可以充分利用计算机硬件的潜力和性能,从而大大提高处理效率,且方法简便、事半功倍。作者最后进一步建议通过利用Parallel Python软件包,可更充分利用计算机多核性能,更大程度提高处理效率。更加满足CERN日益增长的数据快速处理需求。同时本文为类似数据处理需求提供了可借鉴的参考方案。In order to meet CERN＇s （CERN is acronyms of Chinese Ecosystem Research Network） requirements of transforming batch meteorological Excel sheets quickly and loading into database in a high performance and efficiency, The author study and compare several different parallel programming technologies at present, and Python-based parallel programming method is chosen as an approach to transform batch meteorological Excel sheets and load into Oracle database. Firstly the author build an abstract data model to describe data transform structure, then edit code using python-based multiprocessing parallel programming, at last the author analyze different transforming performance under condition of setting various parameters for multiprocessing parallel programming. The study result shows that python-based multiprocessing parallel programming method can fully utilize the computer ＇s hardware potential benchmark so as to promote the transform performance. Moreover, the method is quite simple and high efficient. The author suggest that Parallel Python package should be installed and imported, multi-core computers＇ performance then could be utilized thoroughly to promote CERN＇s data transforming performance much better in the future. This paper also provides a reference solution for other similar data transform requirement.

关 键 词：中国生态系统研究网络 CERN PYTHON 并行编程 多进程 气象 报表 转换入库 

分 类 号：TP31[自动化与计算机技术—计算机软件与理论]