基于分层狄利克雷过程模型的文本分割  

Text Segmentation Based on Hierarchical Dirichlet Processes

作　　者：李天彩 王波[1] 席耀一[1] 张佳明[1] Li Tiancai Wang Bo Xi Yaoyi Zhang Jiaming(Institute of Information and System Engineering, PLA Information Engineering University, Zhengzhou, 450002, Chin)

机构地区：[1]解放军信息工程大学信息系统工程学院,郑州450002

出　　处：《数据采集与处理》2017年第2期408-416,共9页Journal of Data Acquisition and Processing

基　　金：国家高技术研究发展计划("八六三"计划)(2011AA7032030D)资助项目;全军军事研究生课题(2011JY002-158)资助项目

摘　　要：文本分割在文本摘要、信息检索等诸多领域都有重要的应用。主题模型是该领域研究中的重要方法,但目前基于主题模型的方法普遍依赖于主题个数的人工设置。针对此问题,本文提出了一种基于分层狄利克雷过程(Hierarchical Dirichlet process,HDP)模型的文本分割方法。首先使用HDP模型获取文本在主题空间的向量表示,然后将主题向量用于C99分割算法实现文本分割,最后使用两种优化策略对结果进行优化。实验结果表明,基于HDP模型的方法能够摆脱对人工设置主题个数的依赖,有效提高了文本分割的性能。Text segmentation has important applications in many fields,including text summarization,information retrieval,and so on.Topic model is an important tool in text segmentation.However previous text segmentation methods based on topic model generally rely on manually setting of the number of topics influencing results significantly.To solve the problem,a novel text segmentation method based on hierarchical Dirichlet process（HDP）model is proposed.Firstly,texts are modeled with HDP model to get their expression with topic vectors.Then,the topic vectors are used in C99 segmentation algorithm for text segmentation.Finally,two optimization strategies are applied to result optimization.Experimental results show that the presented method can omit manually setting of the topics numbers and improve the performance of text segmentation.

关 键 词：主题模型 文本分割 分层狄利克雷过程 CRF构造 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]