基于HBase/Spark的教学大数据存储及索引模型研究  

A study of the design of a big-data storage and index model for teaching based on HBase/Spark

作　　者：唐立 李亚平[2,3] 曲金帅 TANG Li;LI Ya-ping;QU Jinshuai(Department of Information Engineering,Anbui College of Economie Management,Hefei 230031,China;Teaching Affairs Office,Anhui Collge of Economic Management,Hefei 230031,China;School of Management,Hefei University of Technology,Hefei 230009,China;University of Key Laboratory of Information and Communication on Security Backup and Recovery in Yunnan Province,Yunnan Minzu University,Kunming 650500,China)

机构地区：[1]安徽经济管理学院信息工程系,安徽合肥230031 [2]安徽经济管理学院教务处,安徽合肥230031 [3]合肥工业大学管理学院,安徽合肥230009 [4]云南民族大学云南省高校信息与通信安全灾备重点实验室,云南昆明650500

出　　处：《云南民族大学学报（自然科学版）》2020年第5期486-492,507,共8页Journal of Yunnan Minzu University:Natural Sciences Edition

基　　金：安徽省高校自然科学研究重点项目(KJ2019A0965);安徽继续教育改革项目(2018jxjygg008);云南省应用基础研究计划项目(2018FD055);安徽省教育厅教学研究项目(2019jyxm0910);安徽省高水平教学团队项目(2018jxtd044)。

摘　　要：为解决海量教学数据的高效处理和管理,提出基于HBase/Spark的教学大数据存储及索引模型设计.先基于组合行键构建HBase表,根据教学Course分类进行预分区,并构建cost评分函数,来检测并迁移负载,达到解决数据存储时写热点和负载均衡的问题.语义解析、组合行键索引查询、Spark并行属性条件过滤实现教学大数据高效查询.实验证明基于HBase/Spark的教学大数据的模型,能够实现更加高效的教学大数据访存管理.In order to solve the efficient processing and management problems of massive teaching data,this paper proposes the design of a big-data storage and index model for teaching based on HBase/Spark.First,HBase table is constructed based on the combined row keys,pre-divided according to the teaching course-classification,and a cost-scoring function is constructed to detect and migrate the load so as to solve the problems of writing hot spots and load balancing in data storage.Semantic parsing,the combined row key index query and the Spark parallel attribute condition filter help realize the efficient query of big data for teaching.The experiment proves that this model can realize more efficient retrieval management of the big data for teaching.

关 键 词：HBASE SPARK 组合行键 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论] G434[自动化与计算机技术—计算机科学与技术]