基于文本块密度和标签路径覆盖率的网页正文抽取  

Webpage content extraction via text block density and tag path coverage

作　　者：刘鹏程[1] 胡骏[1] 吴共庆[1] Liu Pengcheng;Hu Jun;Wu Gongqing(School of Computer ＆ Information,Hefei University of Technology,Hefei 230009,Chin)

机构地区：[1]合肥工业大学计算机与信息学院,合肥230009

出　　处：《计算机应用研究》2018年第6期1645-1650,共6页Application Research of Computers

基　　金：国家重点研发计划资助项目(2016YFB1000901);国家自然科学基金资助项目(61273297;61229301;61673152);国家教育部创新团队发展计划资助项目(IRT13059);国家留学基金资助项目(201506695019)

摘　　要：大多数网页除了正文信息外,还包括导航、广告和免责声明等噪声信息。为了提高网页正文抽取的准确性,提出了一种基于文本块密度和标签路径覆盖率的抽取方法(CETD-TPC)。结合网页文本块密度特征和标签路径特征的优点,设计了融合两种特征的新特征,利用新特征抽取网页中的最佳文本块,最后,抽取该文本块中的正文内容。该方法有效地解决了网页正文中噪声块信息过滤和短文本难以抽取的问题,且无须训练和人工处理。在Clean Eval数据集和从知名网站上随机选取的新闻网页数据集上的实验结果表明,CETD-TPC方法在不同数据源上均具有很好的适用性,抽取性能优于CETR、CETD和CEPR算法。Most Webpages contains the content information,as well as noisy information such as navigation,advertisements and disclaimer notices. To address this problem and improve the accuracy of Webpage extraction,this paper proposed a Webpage content extraction method via text block density and tag path coverage（ CETD-TPC）. Combining the advantages of Webpage text block density feature and tag path feature,this paper designed a new feature,TDTPC,which mixed the two features together. Then it extracted the best text block from a Webpage by using the TDTPC feature. Finally,it extracted contents from the content block. Without the manual processing and training,CETD-TPC is an effective solution to deal with the problems of noise block information filtering and short text extraction. Experimental results on Clean Eval datasets and Web news pages randomly selected from well-known websites show that the CETD-TPC method has good applicability on different data sets and performs better than CETR,CETD and CEPR.

关 键 词：正文抽取 文本块密度 标签路径覆盖率 特征融合 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]