A novel unsupervised method for new word extraction  

作　　者：Lili MEI Heyan HUANG Xiaochi WEI Xianling MAO 

机构地区：[1]Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications, Department of Computer Science and Technology,Beijing Institute of Technology

出　　处：《Science China(Information Sciences)》2016年第9期7-17,共11页中国科学（信息科学）（英文版）

基　　金：supported by State Key Program of National Natural Science of China(Grant No.61132009);National High Technology Research and Development Program of China(863 Program)(Grant No.2015AA015404);National Natural Science Foundation of China(Grant Nos.61201351,61402036)

摘　　要：New words could benefit many NLP tasks such as sentence chunking and sentiment analysis. However, automatic new word extraction is a challenging task because new words usually have no fixed language pattern, and even appear with the new meanings of existing words. To tackle these problems, this paper proposes a novel method to extract new words. It not only considers domain specificity, but also combines with multiple statistical language knowledge. First, we perform a filtering algorithm to obtain a candidate list of new words. Then, we employ the statistical language knowledge to extract the top ranked new words. Experimental results show that our proposed method is able to extract a large number of new words both in Chinese and English corpus, and notably outperforms the state-of-the-art methods. Moreover, we also demonstrate our method increases the accuracy of Chinese word segmentation by 10% on corpus containing new words.New words could benefit many NLP tasks such as sentence chunking and sentiment analysis. However, automatic new word extraction is a challenging task because new words usually have no fixed language pattern, and even appear with the new meanings of existing words. To tackle these problems, this paper proposes a novel method to extract new words. It not only considers domain specificity, but also combines with multiple statistical language knowledge. First, we perform a filtering algorithm to obtain a candidate list of new words. Then, we employ the statistical language knowledge to extract the top ranked new words. Experimental results show that our proposed method is able to extract a large number of new words both in Chinese and English corpus, and notably outperforms the state-of-the-art methods. Moreover, we also demonstrate our method increases the accuracy of Chinese word segmentation by 10% on corpus containing new words.

关 键 词：new word extraction word segmentation domain specificity statistical language knowledge domain word extraction 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]