An implementation and optimization for scalable DHT crawler  

作　　者：ZHOU Mo ZHANG JianYu DAI YaFei 

机构地区：[1]The National Laboratory on Local Fiber-Optic Communication Networks and Advanced Optical Communication Systems of Peking University, Beijing 100871, China [2]Institute of Computer Science and Technology of Peking University, Beijing 100871, China

出　　处：《Science China(Information Sciences)》2010年第4期769-779,共11页中国科学（信息科学）（英文版）

基　　金：supported by the National Basic Research Program of China (Grant No. 2004CB318204); the National Natural Science Foundation of China (Grant No. 60873051); the National High-Tech Research & Development Program of China (Grant Nos. 2007AA01Z154, 2006AA01Z410)

摘　　要：KAD is one of the largest scale DHT based on real applications. Measurements on KAD is a good approach for researching DHT. Many different active and passive measurements have been made on those systems, and crawlers are novel approach in active measurement. A crawler begins crawling into the DHT with a basic set of given nodes, sending node searching requests to the nodes in the given set for contact information from more unknown nodes. There are three goals in mind while we design the crawler： finishing crawling the given nodes set as soon as possible; retrieving more nodes information after the crawling; getting result while sending as few network packets as possible. The above goals are correlated with each other. Optimizing one may impact others. This paper proposes a basic DHT crawler framework and discusses possible extension to the framework. After that we exploit the fact that the connectivity in the overlay network is universality, thus we do not need to crawl the whole overlay network space while maintaining the crawling affect.KAD is one of the largest scale DHT based on real applications. Measurements on KAD is a good approach for researching DHT. Many different active and passive measurements have been made on those systems, and crawlers are novel approach in active measurement. A crawler begins crawling into the DHT with a basic set of given nodes, sending node searching requests to the nodes in the given set for contact information from more unknown nodes. There are three goals in mind while we design the crawler： finishing crawling the given nodes set as soon as possible; retrieving more nodes information after the crawling; getting result while sending as few network packets as possible. The above goals are correlated with each other. Optimizing one may impact others. This paper proposes a basic DHT crawler framework and discusses possible extension to the framework. After that we exploit the fact that the connectivity in the overlay network is universality, thus we do not need to crawl the whole overlay network space while maintaining the crawling affect.

关 键 词：DHT CRAWLER network measurements 

分 类 号：TP393[自动化与计算机技术—计算机应用技术] O174.22[自动化与计算机技术—计算机科学与技术]