基于Top-k查询约束的深网增量爬取  

Incremental Deep Web Crawling with Top-k Query Constraint

作　　者：江俊彦[1,2] 彭智勇[1,2] 吴小莹[1] 

机构地区：[1]武汉大学软件工程国家重点实验室,武汉430072 [2]武汉大学计算机学院,武汉430072

出　　处：《模式识别与人工智能》2017年第1期43-53,共11页Pattern Recognition and Artificial Intelligence

基　　金：国家自然科学基金项目(No.61232002;61202035);武汉创新团队计划项目(No.2014070504020237)资助~~

摘　　要：深网数据源的动态性、自治性和体量使第三方应用难以完全爬取所有Web数据.文中研究查询类型(仅允许Top-k查询)和查询资源约束下深网数据源爬取问题,提出基于Top-k查询约束的深网增量爬取方法,结合历史数据和领域知识,优化总体数据质量.首先基于查询树获得有效查询,利用历史数据和领域知识估计查询变化和查询代价.然后,基于估计的查询代价和数据质量,近似选择最优的查询子集最大化总体数据质量.实验表明文中方法较好地提高动态Web数据库爬取的效率和数据质量.Crawling all deep web data is difficult for third party applications due to dynamicity, autonomy and quantity of deep web data sources. To tackle the deep web crawling problem under the query type restriction （only top-k queries are allowed） and limited query resources, an approach for incremental web crawling with top-k query constraint is proposed. Historical data and domain knowledge are combined to maximize total repository data quality. Firstly, valid queries are generated using a query tree, and changes and corresponding cost of the query are estimated by historical data and domain knowledge. Next, grounded on the query cost and data quality of the estimation, the optimal subset is selected approximately to globally maximize total data quality under limited query resources. The experimental results on real datasets show the proposed approach improves the efficiency of crawling dynamic web database.

关 键 词：Top—k查询 Web数据库爬取 数据质量 查询代价 查询选择 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论] TP391.3[自动化与计算机技术—计算机科学与技术]