用于重复充电运营记录的基于块采样的高效聚集查询算法  

Efficient block-based sampling algorithm for aggregation query processing on duplicate charged records

作　　者：潘鸣宇[1] 张禄[1] 龙国标 李香龙[1] 马冬雪[1] 徐亮 PAN Mingyu 1 , ZHANG Lu 1, LONG Guobiao 1, LI Xianglong 1, MA Dongxue 1, XU Liang 2(1. State Grid Beijing Electric Power Company, Beijing 100075, China ; 2. NARI Group, Beijing 102299, Chin)

机构地区：[1]国网北京市电力公司,北京100075 [2]南瑞集团,北京102299

出　　处：《计算机应用》2018年第6期1596-1600,1607,共6页journal of Computer Applications

基　　金：国家电网公司总部科技项目(52020116000j)~~

摘　　要：现有查询分析方法通常将实体识别作为线下预处理过程清洗整个数据集,然而,随着数据规模的不断增大,这种高计算复杂性的线下清洗模式已经很难满足实时性分析应用的需求。针对重复充电运营记录上的聚集查询问题,提出一种将近似聚集查询处理与实体识别相结合的方法。首先,通过基于块的采样策略采集样本;然后,在采集到的样本上利用实体识别方法识别出重复的实体;最后,根据实体识别的结果重构得到聚集结果的无偏估计。所提方法避免了识别全部实体的时间代价,通过识别少量样本数据即可返回满足用户需求的查询结果。真实数据集和合成数据集上的实验结果验证了所提方法的高效性和可靠性。The existing query analysis methods usually treat the entity resolution as an offline preprocessing process to clean the whole data set. However, with the continuous increasing of data size, such offline cleaning mode with high computing complexity has been difficult to meet the needs of real-time analysis in most applications. In order to solve the problem of aggregation query on duplicate charged records, a new method integrating entity resolution with approximate aggregation query processing was proposed. Firstly, a block-based sampling strategy was adopted to collect samples. Then, an entity recognition method was used to identify the duplicate entities on the sampled samples. Finally, the unbiased estimation of aggregated results was reconstructed according to the results of entity recognition. The proposed method avoids the time cost of identifying all entities, and returns the query results that satisfy user needs by identifying only a small number of sample data. The experimental results on both real dataset and synthetic dataset demonstrate the efficiency and reliability of the proposed method.

关 键 词：大数据 实体识别 聚集查询 块采样 分布式计算 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]