海量数据存储模型的研究  

Research on Massive Data Storage Model

作　　者：施磊磊[1] 施化吉[1] 

机构地区：[1]江苏大学计算机科学与通信工程学院,镇江212013

出　　处：《无线通信技术》2014年第4期32-35,40,共5页Wireless Communication Technology

摘　　要：随着互联网上信息量的爆炸式增长,海量网页数据的存储出现了难题。针对海量网页数据进行存储的问题,传统的集中式存储和管理方案已经难以提供高效、可靠和稳定的服务。本文设计并实现了一种针对海量网页数据进行存储的分布式平台模型。该模型利用Hadoop集群和基于HDFS分布式文件系统的Hbase数据库实现高效率地分析、计算和存储海量数据,以MapReduce计算模型和Zookeeper同步协同系统保持数据写入的高效性和一致性。最后通过实验测试,该存储模型可以克服传统的存储模型存储时存在的读写效率低、数据写入不一致的问题,同时具有良好的扩展性、可行性、稳定性和可靠性。With the explosive growth of information on the Internet, mass Webpage data storage ap- peared problem. Aiming at the problem of mass data storage Webpage, centralized storage and man- agement scheme has traditionally difficult to provide efficient, reliable and stable service. In this pa- per, the design and implementation of a data store for the mass Webpage distributed platform model. The model uses the Hadoop cluster and HDFS based distributed file system of Hbase database to re- alize the efficient analysis, calculation and storage of mass data, based on MapReduce computing ef- ficiency and consistency model and Zookeeper synchronous collaborative system to keep the data writing. Finally, through the experimental test, the storage model can overcome the existing tradi- tional storage model when reading and writing efficiency is low, data is written to the inconsistency problem, but also has good scalability, feasibility, stability and reliability.

关 键 词：HADOOP集群 MAPREDUCE HBASE 

分 类 号：TP399[自动化与计算机技术—计算机应用技术]