HSIT:一种针对海量数据的分布式相似性查询索引  

HSIT:Distributed Similarity Query Index for Massive Data

作　　者：姚回 刘文 YAO Hui;LIU Wen(School of Computer Science and Technology,Xinjiang Normal University,Urumqi 830054;School of Control Engineering,Xinjiang Institute of Engineering,Urumqi 830023)

机构地区：[1]新疆师范大学计算机科学技术学院,乌鲁木齐830054 [2]新疆工程学院控制工程学院,乌鲁木齐830023

出　　处：《计算机与数字工程》2025年第3期718-724,共7页Computer & Digital Engineering

基　　金：国家自然科学基金项目(编号:61962058);新疆维吾尔自治区自然科学基金项目(编号:2019D01A30);数据工程与数字矿山联合实验室项目(编号:2019QX0035);新疆维吾尔自治区高校科研计划自然科学项目青年项目(编号:XJEDU2018Y056)资助。

摘　　要：相似性查询常用于信息检索、生物学和网络安全等领域,用来分析数据之间的关联关系。传统方法执行相似性查询往往需要查询点与数据库中的每一条数据进行计算。随着数据量的增大,计算量会成线性式增长。为提升海量数据的分布式相似性查询效率,提出一种基于HBase的相似性查询索引结构HSIT(HBase Similarity Index Tree)。在数据存储的过程中,该算法实现动态建立相似性索引树结构。HSIT索引能够按照相似度阈值,划分相似性的数据在HBase的相邻区域存储;在用户执行相似性查询时,查询节点可以通过HSIT快速检索相似区域。该索引能够实现高效剪枝,使得只有相似的区域才需要两两计算。通过2万条数据指数型增长到128万条数据执行相似性查询,与DSCS-LTS算法比较,实验结果证明,HSIT算法效率有所提升。Similarity queries are commonly used in fields such as information retrieval,biology,and network security to analyze associations between data.The traditional method to perform similarity query often requires the query point and each piece of data in the database to be calculated.As the amount of data increases,the amount of computation increases linearly.In order to improve the efficiency of distributed similarity query of massive data,a HBase similarity index tree(HSIT)is proposed.In the process of data storage,the algorithm realizes the dynamic establishment of similarity index tree structure.The HSIT index can divide the similarity data according to the similarity threshold and store it in the adjacent area of HBase.When a user performs a similarity query,the query node can quickly retrieve similar regions through HSIT.The index can achieve efficient pruning,so that only similar regions need to be calculated in pairs.The similarity query is performed through the exponential growth of 20000 data to 1.28 million data.Compared with the DSCS-LTS algorithm,the experimental results show that the efficiency of the HSIT algorithm has been improved.

关 键 词：海量数据 分布式 相似性查询 索引 HBASE 

分 类 号：P413[天文地球—大气科学及气象学]