一种基于状态转换图的Ajax爬行算法  

Ajax crawling algorithm based on state transition graph

作　　者：郭浩[1] 陆余良[1] 刘金红[1] 

机构地区：[1]电子工程学院网络系,合肥230037

出　　处：《计算机应用研究》2009年第11期4266-4269,共4页Application Research of Computers

摘　　要：传统Web爬虫无法解决爬行Ajax应用所面临的JavaScript执行、状态识别与切换、重复状态检测等问题。为此,首先定义Ajax应用的状态转换图,并设计了一种基于状态转换图的Ajax爬行算法,通过该算法可以获取Ajax应用状态信息和后台DeepWeb资源。为了提高Ajax爬行的准确性、减少待爬行的状态数目,使用Ajax指纹识别、DOM结构过滤等方法改进上述算法。实验结果表明了算法的有效性和性能。Traditional Web crawler could not meet the challenges of crawling Ajax application, such as JavaScript execution, state identification and navigation, duplicate states elimination etc. By exploring such challenges, this paper introduced state transition graph, based on which an algorithm was proposed to retrieve Ajax states and the background Deep Web. In order to uplift the accuracy, reduce the unnecessary states, improved the algorithm by Ajax fingerprinting and DOM filtering. The experimental results indicate the effectivity and efficiency of this algorithm.

关 键 词：Ajax爬虫 状态转换图 WEB爬虫 DEEP WEB 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]