基于WSFT模型的深层网文本获取方法  

Approach based on WSFT for crawling deep web

作　　者：杨贯中 李虹萱 YANG Guanzhong;LI Hongxuan(School of Information Science and Engineering, Hunan University, Changsha 410082, China)

机构地区：[1]湖南大学信息科学与工程学院,长沙410082

出　　处：《计算机工程与应用》2017年第18期236-242,共7页Computer Engineering and Applications

基　　金：湖南省产学研结合重大科技成果转化项目(No.2010XK6024)

摘　　要：Ajax技术在深层网(Deep Web)网站开发中得到了广泛应用。针对Ajax页面多状态、状态之间强关联的特性,提出一种构建WSFT(带权状态融合树)模型的方法,来进行Ajax页面文本信息预处理。引入了文本特征树作为状态指纹进行状态捕获,优化了当前Ajax页面数据采集方法,同时通过StatusRank方法计算状态转移权值来分析状态迁移信息,最后生成WSFT。实验证明,该方法能有效地获取Ajax页面多状态文本信息,并且有助于后续Web挖掘的重要文本内容提取。Ajax technology has been widely applied in deep web application development.This paper constructs a WeightedState Fusion Tree(WSFT)model to pre-process the text information in web page with Ajax technology which has multiplestates with strong correlation.Firstly,the current approach of Ajax page data collection is optimized by regarding textfeature tree as a fingerprint to traverse through the multiple states.Secondly,the transition weight with StatusRank methodis calculated for each states of the Ajax page.The state transition information is analyzed.Finally,a WSFT is generated.The experimental results show that the proposed method can effectively obtain the text information in Ajax page with multiplestates,and help the follow-up important text extraction of web mining.

关 键 词：Ajax爬虫 带权状态融合树 文本挖掘 文本特征树 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]