基于爬虫和网站分类的主题信息源发现方法  

Discovery of topic-specific information source based on web crawler and website classification

作　　者：邓厚平[1] 武刚[1] 

机构地区：[1]北京林业大学信息学院,北京100083

出　　处：《计算机工程与应用》2016年第3期59-65,共7页Computer Engineering and Applications

基　　金：中央高校基本科研业务费专项基金资助项目(No.BLYX200928)

摘　　要：如何发现主题信息源是主题Web信息整合的前提。提出了一种主题信息源发现方法,将主题信息源发现转化为网站主题分类问题,并利用站外链接发现新的信息源。从网站中提取出能反映网站主题的内容特征词和结构特征词,建立描述网站主题的改进的向量空间模型。以该模型为基础,通过类中心向量法与SVM相结合对网站主题进行分类。提出一种能尽量少爬取网页的网络搜索策略,在发现站外链接的同时爬取最能代表网站主题的页面。将该主题信息源发现方法应用于林业商务信息源,通过实验验证了该方法的有效性。The discovery of topic-specific information source is the premise of Web information integration. A topic-specific information discovery method is presented, changing the problem to website topic classification and discover websites using external links. An improved VSM model is established to describe the website topic, using both content and structure features extracted from websites. Based on the improved VSM model, a classification method combining center-vector algorithm and SVM is presented to classify the topic of websites. A web search strategy aiming to minimize the quantity of crawled web page is presented to find out web pages that best represent the topic of the website. The topic-specific information source discovery method is used to find forestry business website for test and performs well.

关 键 词：网站主题 特征描述 分类 爬虫 信息源发现 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]