基于文本特征值的正文抽取方法  

Web content extraction method based on text feature value

作　　者：孟川[1] 武小年[1,2] 

机构地区：[1]桂林电子科技大学信息与通信学院,广西桂林541004 [2]桂林电子科技大学广西可信软件重点实验室,广西桂林541004

出　　处：《桂林电子科技大学学报》2017年第2期106-110,共5页Journal of Guilin University of Electronic Technology

基　　金：广西自然科学基金(2015GXNSFGA139007);广西无线宽带通信与信号处理重点实验室基金(GXKL061510;GXKL0614110);广西可信软件重点实验室基金(KX201622);桂林电子科技大学研究生教育创新计划(YJCXS201524)

摘　　要：针对现有Web正文抽取方法适用性差、准确率低的问题,提出了一种基于文本特征值的正文抽取方法。该方法通过对Web页面的代码进行预处理,再解析转换成DOM树;通过遍历DOM树,以节点的文本长度和标点符号权重计算DOM树各节点的文本特征值,并通过标准差消除噪声;通过高斯函数为节点的文本特征值进行平滑处理,缓解节点文本特征值的突变,降低短文本节点丢失的可能。实验测试结果表明,该方法不依赖标签,也无需训练数据,具有较好的通用性和较高的正文抽取准确率。In view of poor universality and low accuracy of the existing Web text extraction methods, a text extraction method based on text feature value is proposed. Firstly codes of Web pages are preprocessed, and the preproeessed codes are conver- ted into the DOM tree. Then through traversing the DOM tree, the text feature value of each DOM tree node is calculated based on the text length and punctuation weight of node, and the standard deviation is used to eliminate noise as much as possible. Gauss function is used to smooth the text feature value of nodes, ease the mutation of text feature value, and even- tually reduce the possible loss of short text node. The experimental results show that the presented method does not rely on the label, need not training data, and has good versatility and high accuracy.

关 键 词：正文抽取 主题网页 文本特征值 高斯平滑 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]