CC-MRSJ:Hadoop平台下缓存敏感的星型联接算法  

CC-MRSJ:Cache Conscious Star Join Algorithm on Hadoop Platform

作　　者：周国亮[1,2] 朱永利[1] 王桂兰[1] 

机构地区：[1]华北电力大学控制与计算机工程学院,保定071003 [2]国网冀北电力有限公司技能培训中心,保定071051

出　　处：《电信科学》2013年第10期31-37,共7页Telecommunications Science

基　　金：中央高校基本科研业务费专项基金资助项目(No.13MS103);河北省高等学校科学研究基金资助项目(No.Z2011306)

摘　　要：提出了一种缓存敏感的MapReduce星型联接算法,事实表每列单独存储,维表根据维层次划分为多个列簇。事实表外键列与对应维表采用相关性存储,减少联接过程中的数据移动。算法分为两个阶段,首先每个外键列和对应维表进行联接;然后对中间结果进行联接,随机访问测度列,进而得到最终结果。算法只读取需要的数据,缓存利用率高,从而具有良好的缓存敏感特性;算法充分利用时延实体化,避免不必要的数据访问和移动。通过在SSB数据集上与Hive系统的对比测试表明,CC-MRSJ算法具有较高的执行效率。A cache-conscious MapReduce star join algorithm was presented,each column of fact table was separately stored,and dimension table was divided into several column families according to dimension hierarchy.Fact table foreign key column and corresponding dimension table was co-location storage,thus reducing data movement in the join process.CC-MRSJ consists of two phases：firsdy each foreign key column and the corresponding dimension table were joined; then the intermediate results were joined and random accessed measure columns,and so got the final result.CC-MRSJ read only the data needed,and cache utilization is high,so it has good cache conscious feature; it also takes advantage of late materialization,avoiding unnecessary data access and movement.CC-MRSJ has higher performance comparing to hive system based on SSB datasets.

关 键 词：星型联接 MAPREDUCE 缓存敏感 存储模型 

分 类 号：TP212[自动化与计算机技术—检测技术与自动化装置]