国家图书馆WEB数据增量采集设计及其实现  

Design and Implementation on the Web Data Deduplicated Crawlers of the National Library of China

作　　者：季士妍[1] 赵丹阳 JI ShiYan;ZHAO DanYang(National Library of China,Beijing 100081,China)

机构地区：[1]国家图书馆,北京100081

出　　处：《数字图书馆论坛》2021年第1期32-37,共6页Digital Library Forum

摘　　要：本文详细介绍网络资源保存技术策略现状,并从国家图书馆网络资源采集的实际业务需求出发,制定并设计符合国家图书馆业务需求的增量采集技术策略,简述国家图书馆基于Heritrix3.4的增量采集实现方法和实验效果,以期为业界提供有益的参考和借鉴。This paper introduces the current situation of web archiving technology strategy in detail,and designs the deduplicated crawlers technology strategy based on the actual practices of web archiving in the National Library of China.It describes the realization method of duplicated crawlers based on heritrix 3.4,so as to provide useful reference for the industry.

关 键 词：国家图书馆 增量采集 HERITRIX 

分 类 号：G255[文化科学—图书馆学]