一种高速URL过滤算法的研究与应用  

Research and Application of a High Speed URL Filtering Algorithm

作　　者：黄诚[1] 

机构地区：[1]四川大学计算机学院,成都610065

出　　处：《现代计算机》2016年第2期13-16,共4页Modern Computer

摘　　要：当前,传统防火墙的URL过滤方式只是对于规则库中的URL进行过滤,对于新增的涉黄涉暴网站无能为力,或者管理员响应迟钝。针对当前这种现状,提出一种局域网内URL过滤系统,基于网络爬虫和敏感词过滤技术通过爬去网页文本和对于网页文本分析来判断指定URL是否合法。考虑到匹配效率和本过滤系统所使用的内存空间,使用MD5对URL计算摘要值,在此之上建立黑白名单,再结合Bloom Filter算法和改进的Hash表数据结构用以实现对URL的高速过滤。Recently, traditional URL filtering firewall rule base only for URL filtering, for the new added website involving violence powerless, or the administrator unresponsive. For this view of the eurrent situation, proposes a URL filtering system within a local area network, which is based on climbing web pages for text and analyzing text to determine the lawfulness of the specified URL, considering the matching efficiency of the words and the use of memory space in this system, uses the MD5 digest value calculated on the URL, builds on top of this black and white lists, combining Bloom Filter algorithm with improved HashMap data structure to achieve high speed for URL filtering.

关 键 词：URL过滤 网络爬虫 敏感词过滤 BLOOM Filter HASH表 MD5 

分 类 号：TP391.3[自动化与计算机技术—计算机应用技术]