Probabilistic, Statistical and Algorithmic Aspects of the Similarity of Texts and Application to Gospels Comparison  

作　　者：Soumaila Dembele Gane Samb Lo 

机构地区：[1]LSTA, Université Pierre et Marie Curie, Paris, France [2]LERSTAD, Université Gaston Berger de Saint-Louis, Saint-Louis, Sénégal [3]Université des Sciences de Gestion de Bamako, Bamako, Mali

出　　处：《Journal of Data Analysis and Information Processing》2015年第4期112-127,共16页数据分析和信息处理（英文）

摘　　要：The fundamental problem of similarity studies, in the frame of data-mining, is to examine and detect similar items in articles, papers, and books with huge sizes. In this paper, we are interested in the probabilistic, and the statistical and the algorithmic aspects in studies of texts. We will be using the approach of k-shinglings, a k-shingling being defined as a sequence of k consecutive characters that are extracted from a text (k ≥ 1). The main stake in this field is to find accurate and quick algorithms to compute the similarity in short times. This will be achieved in using approximation methods. The first approximation method is statistical and, is based on the theorem of Glivenko-Cantelli. The second is the banding technique. And the third concerns a modification of the algorithm proposed by Rajaraman et al. ([1]), denoted here as (RUM). The Jaccard index is the one being used in this paper. We finally illustrate these results of the paper on the four Gospels. The results are very conclusive.The fundamental problem of similarity studies, in the frame of data-mining, is to examine and detect similar items in articles, papers, and books with huge sizes. In this paper, we are interested in the probabilistic, and the statistical and the algorithmic aspects in studies of texts. We will be using the approach of k-shinglings, a k-shingling being defined as a sequence of k consecutive characters that are extracted from a text (k ≥ 1). The main stake in this field is to find accurate and quick algorithms to compute the similarity in short times. This will be achieved in using approximation methods. The first approximation method is statistical and, is based on the theorem of Glivenko-Cantelli. The second is the banding technique. And the third concerns a modification of the algorithm proposed by Rajaraman et al. ([1]), denoted here as (RUM). The Jaccard index is the one being used in this paper. We finally illustrate these results of the paper on the four Gospels. The results are very conclusive.

关 键 词：SIMILARITY Web MINING Jaccard SIMILARITY RU Algorithm Minhashing Data MINING Shingling Bible’s GOSPELS Glivenko-Cantelli EXPECTED SIMILARITY STATISTICAL Estimation 

分 类 号：R73[医药卫生—肿瘤]