基于共享缓冲区的数据流处理框架设计与实现  

Design and implementation of data stream processing framework based on shared buffer

作　　者：何小东[1] 尹海波[1] 

机构地区：[1]中南林业科技大学计算机与信息工程学院,湖南长沙410003

出　　处：《计算机工程与设计》2012年第11期4398-4401,4406,共5页Computer Engineering and Design

基　　金：湖南省自然科学基金项目(10JJ5066)

摘　　要：为解决数据流处理应用程序依赖数据流传输,分布式环境部署困难,应用开发难度大等问题,设计了一种基于共享缓冲区的数据流处理框架。该框架利用共享缓冲区为应用提供数据代理接口,实现数据流推动和分布式处理,并引入可自定义的负载均衡,解决了应用对数据流的依赖度和开发部署的难度系数,使其专注数据分析处理,无需关心数据的地理位置,提高了应用系统的稳定性和扩展性。通过在互联网病毒监控系统中应用,验证了该框架在分布式环境下的实时性和灵活性,使其可移植到云计算环境中。To solve the data transfer dependence and to decrease coefficient of difficulty in the deployment of the distributed environment and the development, a data stream processing framework base on shared buffer is proposed for data process processing application. This design using a shared buffer proxy provides data for application, push data among nodes and process in distributed manner. User define load balancing is introduced. A data dependence for application is solved, and makes it realized data analysis and processing, without concern for the geographic location of the data. The stability and scalability of the application system is improved. Finally, the real-time and flexibility of the framework in a distributed environment is validated by realizing the Internet virus monitoring system. So that it can be ported to a cloud computing environment.

关 键 词：流数据处理 数据处理 共享缓冲 分布式 云计算 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]